{"id":20685494,"url":"https://github.com/yas-sim/openvino-model-server-wrapper","last_synced_at":"2025-08-01T18:13:42.195Z","repository":{"id":185235192,"uuid":"443766694","full_name":"yas-sim/openvino-model-server-wrapper","owner":"yas-sim","description":"Python wrapper class for OpenVINO Model Server. User can submit inference request to OVMS with just a few lines of code.","archived":false,"fork":false,"pushed_at":"2022-01-16T08:28:02.000Z","size":25644,"stargazers_count":9,"open_issues_count":2,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T13:52:29.628Z","etag":null,"topics":["ai","area-intrusion-detection","cloud","deep-learning","edge","grpc","grpc-client","inference","intel","line-crossing-detection","model-serving","object-tracking","openvino","openvino-docker","openvino-model-server","python","serving","tensorflow-serving","triton-inference-server"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yas-sim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-01-02T13:00:07.000Z","updated_at":"2024-07-15T02:25:04.000Z","dependencies_parsed_at":"2023-08-01T07:09:29.929Z","dependency_job_id":null,"html_url":"https://github.com/yas-sim/openvino-model-server-wrapper","commit_stats":null,"previous_names":["yas-sim/openvino-model-server-wrapper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yas-sim/openvino-model-server-wrapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-model-server-wrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-model-server-wrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-model-server-wrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-model-server-wrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yas-sim","download_url":"https://codeload.github.com/yas-sim/openvino-model-server-wrapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yas-sim%2Fopenvino-model-server-wrapper/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268273242,"owners_count":24223791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","area-intrusion-detection","cloud","deep-learning","edge","grpc","grpc-client","inference","intel","line-crossing-detection","model-serving","object-tracking","openvino","openvino-docker","openvino-model-server","python","serving","tensorflow-serving","triton-inference-server"],"created_at":"2024-11-16T22:27:34.393Z","updated_at":"2025-08-01T18:13:42.166Z","avatar_url":"https://github.com/yas-sim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# OpenVINO Model Server wrapper API for Python\n## Description\nThis project provides a Python wrapper class for [OpenVINO Model Server](https://github.com/openvinotoolkit/model_server) ('OVMS' in short).  \nUser can submit DL inference request to OVMS with just a few lines of code.  \n\nThis project also includes the [instruction to setup OpenVINO model server to support multiple models](#how-to-setup-openvino-model-server-for-multiple-model-support-ubuntu).\n\nAlso, the project provides [an automation Python script to generate OVMS model repository](#openvino-model-server---model-repository-setup-automation-tool) in a single line of command. Users can generate a model repository for OVMS by just  preparing a directory which contains multiple OpenVINO IR models, and running the script.\n\nThis wrapper API needs 'tensorflow' and 'tensorflow-serving-api' to run.  \nFor the users those don't want to install those big libraries, **this project provides an [alternative solution](#how-to-run-ovms-wrapper-api-without-having-tensorflow-and-tensorflow-serving-api)**. User can use bundled gRPC handler code in this project instead of installing TensorFlow and TensorFlow-serving-API. This alternative solution allows user to submit inference request to OVMS from non-IA client devices.  \n\nThe project also includes several [demo programs](#demo-programs) as user's reference.  \n\n![hp](./samples/human-pose-estimation-2d/resources/human-pose-demo.png)\n![od](./samples/object-tracking-line-crossing-area-intrusion/resources/object-track.gif)\n\n## Sample client inference code (Classification - ResNet-50)\n```python\nfrom ovms_wrapper.ovms_wrapper import OpenVINO_Model_Server\n\nimport cv2\nimport numpy as np\n\novms = OpenVINO_Model_Server()              # Create an OVMS class instance\novms.connect('127.0.0.1', 9000)             # Connect to an OVMS on 127.0.0.1:9000\n#print(ovms.get_model_status('resnet_50'))\nmodel = ovms.open_model('resnet_50')        # Open a model on OVMS\nprint(model.inputs, model.outputs)          # Display input/output blob info\n\n# curl -O https://raw.githubusercontent.com/intel-iot-devkit/smart-video-workshop/master/Labs/daisy.jpg\nimage_file  = 'daisy.jpg'\nimg = cv2.imread(image_file)                # Read an image\nres = model.single_image_infer(img)         # Infer\nresult = res[model.outputs[0].name]         # Retrieve the infer result\n\n# display result\nnu = np.array(result)\nma = np.argmax(nu)\nprint(\"Result:\", ma)\n```\n\n## Prerequisites\nYou need to install some Python modules.   \n```sh\npython3 -m pip install --update pip setuptools\npython3 -m pip install -r requirements.txt\n```\n----\n## How to setup OpenVINO Model Server (Single model support, Ubuntu)  \nWhen you just need only single IR model to be served by OVMS, you don't need to have model repository. Just specify the model directory and give some information such as model name as option to start the OVMS docker container.   \nThis is the easiest way to start OVMS.  \nNote: OVMS can run on Windows too. Please refer to the [official OVMS document](https://docs.openvino.ai/latest/openvino_docs_ovms.html) for details.  \n1. Install prerequisites  \n```sh\nsudo apt update \u0026\u0026 sudo apt install -y docker.io python3-venv\npython3 -m pip install --upgrade pip setuptools\npython3 -m pip install tensorflow tensorflow-serving-api\n```\n2. Prepare a DL model  \nInstall OpenVINO for temporal use, and download and convert a model.  \n**Note:** You don't need to re-create a new 'venv' if you already have one. Just activate it and use it.  \n```sh\npython3 -m venv venv__temp\n. venv__temp/bin/activate\npython3 -m pip install openvino-dev tensorflow\nomz_downloader --name resnet-50-tf\nomz_converter  --name resnet-50-tf --precisions FP16\ndeactivate\n```\n3. Start OpenVINO Model Server as a Docker container  \nDocker will pull the pre-built '`openvino/model_server`' image from the Docker hub, create a container and run it.  \n```sh\ndocker run -d --rm --name ovms \\\n  -v $PWD/public/resnet-50-tf/FP16:/models/resnet50/1 \\\n  -p 9000:9000 \\\n  openvino/model_server:latest \\\n  --model_path /models/resnet50 \\\n  --model_name resnet_50 \\\n  --port 9000\n```\nOVMS will start serving the Resnet-50 model as model-name='resnet_50', model-version=1, and gRPC-port=9000.  \nNow you can run a sample client inference program to test the OVMS.  \n**Note:** You can run the client code from any PC or node as long as it is IP reachable to the OVMS. You might need to modify the IP address in the client code accordingly.  \n\n----\n## How to setup OpenVINO Model Server for multiple model support (Ubuntu)  \nOVMS requires a model repository which contains the IR models when you want to support multiple models. The repository must follow strict directory and file structure. Also, you need to create a model configuration file in JSON format.  \n### Prepare the model directory with IR models and configuration files (JSON)  \n1. Install prerequisites  \n```sh\nsudo apt update \u0026\u0026 sudo apt install -y docker.io python3-venv\npython3 -m pip install --upgrade pip setuptools\npython3 -m pip install tensorflow tensorflow-serving-api\n```\n2. Prepare DL models  \nInstall OpenVINO for temporal use, and download and convert the models.  \n**Note1:** Following steps are summarized in `'setup_model_repository.[sh|bat]`' shell script for user's convenience.  \n**Note2:** You don't need to re-create a new 'venv' if you already have one. Just activate it and use it.  \n**Note3:** '`face-detection-0200`' model is a Intel model. It is distributed as an OpenVINO IR model. You can use the model by just downloading it without conversion.   \n```sh\npython3 -m venv venv__temp\n. venv__temp/bin/activate\npython3 -m pip install openvino-dev tensorflow\nomz_downloader --name resnet-50-tf,googlenet-v1-tf,face-detection-0200 --precisions FP16\nomz_converter  --name resnet-50-tf,googlenet-v1-tf                     --precisions FP16\ndeactivate\n```\n\n3. Setup the model repository for OVMS.  \nOVMS requires the IR models to be stored in a specific directory structure. You need to create a compatible directory tree structure and place IR models accordingly. Also, OVMS requires a repository configuration file (`config.json`). Please refer to the [official document](https://github.com/openvinotoolkit/model_server/blob/main/docs/docker_container.md#configfile) for details.  \n**Note1:** `config.json` defines model specification in the model repository.  \n**Note2:** `mapping_config.json` defines alias name for input and output blobs in the model. You can give fiendly name to those blobs for your convenience. This is optional.  \n```sh\nmkdir -p ./ovms_model_repository/models/resnet-50-tf/1\nmkdir -p ./ovms_model_repository/models/googlenet-v1-tf/1\nmkdir -p ./ovms_model_repository/models/face-detection-0200/1\ncp ./public/resnet-50-tf/FP16/*       ./ovms_model_repository/models/resnet-50-tf/1/\ncp ./public/googlenet-v1-tf/FP16/*    ./ovms_model_repository/models/googlenet-v1-tf/1/\ncp ./intel/face-detection-0200/FP16/* ./ovms_model_repository/models/face-detection-0200/1/\ncp ./model-config.json                ./ovms_model_repository/models/config.json\ncp ./mapping_config-resnet-50-tf.json ./ovms_model_repository/models/resnet-50-tf/1/mapping_config.json\n```\n\n- The model repository directory structure after this operation will look like this.  \n```\novms_model_repository/\n└── models\n    ├── config.json                     # \u003c- model configuration file\n    ├── face-detection-0200\n    │   └── 1                           # \u003c- Model version number. A positive integer value\n    │       ├── face-detection-0200.bin\n    │       └── face-detection-0200.xml\n    ├── googlenet-v1-tf\n    │   └── 1\n    │       ├── googlenet-v1-tf.bin\n    │       ├── googlenet-v1-tf.mapping\n    │       └── googlenet-v1-tf.xml\n    └── resnet-50-tf\n        └── 1\n            ├── mapping_config.json     # \u003c- in/out blob name alias definition. optional\n            ├── resnet-50-tf.bin\n            ├── resnet-50-tf.mapping\n            └── resnet-50-tf.xml\n```\n- '`config.json`' file contains the model specifications. You can specify OpenVINO plugin options ('`plugin_config`'), target device ('`target_device`') and so on.  \n**Note:** You need to use a specific Docker image if you want to use the integrated GPU. The Docker image name should be '`openvino/model-server:latest-gpu`'.  \n```json\n{\n    \"model_config_list\":[\n        {\n            \"config\": {\n                \"name\":\"resnet_50\",       # \u003c- Model name which will be exposed to the clients\n                \"base_path\":\"/opt/models/resnet-50-tf\",\n                \"batch_size\":\"1\",\n                \"plugin_config\": {\"CPU_THROUGHPUT_STREAMS\": \"CPU_THROUGHPUT_AUTO\"}\n            }\n        },\n        {\n            \"config\": {\n                \"name\":\"googlenet_v1\",\n                \"base_path\":\"/opt/models/googlenet-v1-tf\",\n                \"batch_size\":\"1\",\n                \"nireq\":4,\n                \"target_device\":\"CPU\"\n            }\n        },\n        {\n            \"config\": {\n                \"name\":\"face-detection-0200\",\n                \"base_path\":\"/opt/models/face-detection-0200\"\n            }\n        }\n    ]\n}\n\n```\n\n4. Start OVMS Docker container with the model repository.  \nDocker will pull the pre-built '`openvino/model_server`' image from the Docker hub, create a container and run it.  \n```sh\ndocker run -d --rm --name ovms \\\n  -v $PWD/ovms_model_repository/models:/opt/models \\\n  -p 9000:9000 openvino/model_server:latest \\\n  --config_path=/opt/models/config.json \\\n  --port 9000\n```\n\n- If you want to use the integrated GPU, you need to use a specific Docker image with '`latest-gpu`' tag and add `'--device=/dev/dri`' option when you start the container. Please refer to the [officiel document](https://github.com/openvinotoolkit/model_server/blob/main/docs/docker_container.md#running-openvino-model-server-with-ai-accelerators-ncs-hddl-and-gpu-) for the datails.  \n```sh\ndocker run -d --rm --name ovms \\\n  -v $PWD/ovms_model_repository/models:/opt/models \\\n  -p 9000:9000 openvino/model_server:latest-gpu \\\n  --device=/dev/dri \\\n  --config_path=/opt/models/config.json \\\n  --port 9000\n```\n\nNow OVMS serves '`resnet_50`', '`googlenet_v1`' and '`face-detection-0200`' models.\n\n----\n\n## OpenVINO Model Server - Model repository setup automation tool  \n`setup_ovms_model_repo.py` in `./model-repo-generator/` searches OpenVINO IR models in the specified source directory and create an model repository for OpenVINO Model Server. It generates required `config.json` file as well.  \nUser can create the model repository with this script and just pass it to OVMS to start the inference service.   \n\n|option|descriotion|\n|----|----|\n|`-m`, `-model_dir`| Source directory that contains OpenVINO IR models. Required.|\n|`-o`, `-output_dir`| OVMS model repository directory to generate. Default='`./model_repository`'|\n|`--verbose`|Verbose mode flag. Optional. Default=False|\n|`--dryrun`|Dryrun flag. Nothing will be written nor generated if this flag is set. Useful with `--verbose` flag. Optional. Default=False|\n- Command line example:\n```sh\npython3 model-repo-generator/setup_ovms_model_repo.py -m ir_models -o ovms_repo\n```\n![image](resources/repo-generation.png)\n\n## How to run OVMS wrapper API without having TensorFlow and tensorflow-serving-api  \nTensorFlow and its dependency libraries take up about 2GB of storage space. You may not want to install TF in case the target device is a kind of small and not-powerful device such as low-power ARM based devices.  \nThis project offers an alternative solution to resolve this issue. You can use the bundled gRPC handler code instead of TF and TF-serving-api. With those gRPC handler code, it is not necessary to install TF and TF-serving-API to run the OVMS client codes.     \n1. Rename directories  \nRemove '`_`' on top of '`_tensorflow`' and '`_tensorflow_serving`' directory names.  \nThis operation would make a name space conflict of Python modules if you have already installed 'tensorflow' and 'tensorflow-serving-api' on your system. You must not have those Python modules.   \n```bash\nmv _tensorflow tensorflow\nmv _tensorflow_serving tensorflow_serving\n```\nNow you can use the OVMS wrapper without having 'tensorflow' and 'tensorflow-serving-api'.   \n\n**Note:** You can re-generate the gRPC handler codes from `.proto` files by running '`build_proto.sh`'.\n```bash\n./build_proto.sh\n```\n- You should have following directory structure to run your code properly.  \n```\n.\n├── ovms_wrapper\n├── tensorflow\n├── tensorflow_serving\n└── \u003cYOUR_CODE.py\u003e\n```\n**Exmple:** Running OVMS client program using OVMS wrapper on Raspbian OS (without Tensorflow installed)  \n![raspberrypi](resources/raspberrypi.png)\n\n## Demo Programs  \nYou can try demo programs with OVMS wrapper.  \nYou need to download required models and start OVMS before you run the demo programs.\n\n1. Setup OVMS  \n[How to Setup OVMS for Demo Programs](./samples/ovms_setup_for_demos/README.md)\n\n2. Human Pose Estimation demo  \nYou need to build a C++ based Python module to run demo.  \n[How to run](./samples/human-pose-estimation-2d/README.md)  \n![hp](./samples/human-pose-estimation-2d/resources/human-pose-demo.png)\n\n3. Object Detection / Line Crossing / Area Intrusion demo  \n[How to run](./samples/object-tracking-line-crossing-area-intrusion/README.md)  \n![od](./samples/object-tracking-line-crossing-area-intrusion/resources/object-track.gif)\n\nEND\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino-model-server-wrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyas-sim%2Fopenvino-model-server-wrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyas-sim%2Fopenvino-model-server-wrapper/lists"}