{"id":25917108,"url":"https://github.com/codycollier/tfs-model-status-probe","last_synced_at":"2025-07-12T13:35:41.655Z","repository":{"id":42676672,"uuid":"315506704","full_name":"codycollier/tfs-model-status-probe","owner":"codycollier","description":"A kubernetes probe which checks model status for a TensorFlow Serving model","archived":false,"fork":false,"pushed_at":"2022-07-08T14:30:07.000Z","size":72,"stargazers_count":17,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-15T07:45:55.387Z","etag":null,"topics":["grpc","kubernetes","tensorflow","tensorflow-serving"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/codycollier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-11-24T03:21:41.000Z","updated_at":"2024-09-25T23:45:14.000Z","dependencies_parsed_at":"2022-09-11T23:22:41.890Z","dependency_job_id":null,"html_url":"https://github.com/codycollier/tfs-model-status-probe","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codycollier%2Ftfs-model-status-probe","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codycollier%2Ftfs-model-status-probe/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codycollier%2Ftfs-model-status-probe/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/codycollier%2Ftfs-model-status-probe/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/codycollier","download_url":"https://codeload.github.com/codycollier/tfs-model-status-probe/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241670112,"owners_count":20000327,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grpc","kubernetes","tensorflow","tensorflow-serving"],"created_at":"2025-03-03T13:19:52.331Z","updated_at":"2025-03-03T13:19:53.084Z","avatar_url":"https://github.com/codycollier.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## tfs_model_status_probe - TensorFlow Serving Model Status Probe\n\n![ci](https://github.com/codycollier/tfs-model-status-probe/workflows/ci/badge.svg)\n![release](https://github.com/codycollier/tfs-model-status-probe/workflows/release/badge.svg)\n\n\n\nThe `tfs_model_status_probe` checks the model status in a TensorFlow Serving instance [1].  The probe is modeled after `grpc_health_probe` [2] and is intended for use as a kubernetes probe.\n\nThe probe calls the ModelService.GetModelStatus() rpc [3] for a given model.  If the model is `AVAILABLE`, then the probe will have an exit code of 0.  If the model is still `LOADING`, in some other state, or there are grpc communication errors, then the exit code will be non-zero.  If no version is provided, the probe assumes a single version and only checks the first version in the response.\n\n\n#### Usage\n\n```\n$ ./tfs_model_status_probe -help\nUsage of ./tfs_model_status_probe:\n  -addr string\n    \tThe hostname:port to check (default \"localhost:9000\")\n  -connect-timeout duration\n    \tTimeout for making connection (default 3s)\n  -model-name string\n    \tThe name of the model (default \"default\")\n  -model-version int\n    \tThe version of the model\n  -rpc-timeout duration\n    \tTimeout for rpc call (default 10s)\n```\n\n\n## Examples\n\nHere are a handful of the more common examples of success and failure calls.\n\n\nSuccessful call with model status AVAILABLE (exit code 0):\n```\n$ ./tfs_model_status_probe -addr=\"localhost:8500\" -model-name=\"half_plus_two\"\n2020/11/30 19:49:33 ModelStatusResponse: model_version_status:{version:123 state:AVAILABLE status:{}}\n2020/11/30 19:49:33 Servable state is AVAILABLE\n\n$ echo $?\n0\n```\n\n\nError calling with wrong model name (exit code 10):\n```\n$ ./tfs_model_status_probe -addr=\"localhost:8500\" -model-name=\"no-such-model\"\n2020/11/30 19:51:27 ModelStatusResponse: \u003cnil\u003e\n2020/11/30 19:51:27 Model not found: rpc error: code = NotFound desc = Could not find any versions of model no-such-model\n\n$ echo $?\n10\n```\n\n\nError when unable to communicate with service (exit code 2):\n```\n$ ./tfs_model_status_probe -addr=\"localhost:1234\" -model-name=\"half_plus_two\"\n2020/11/30 19:52:57 Error dialing grpc service: context deadline exceeded\n\n$ echo $?\n2\n```\n\n\nThe examples are run using a test server like so:\n```\n$ git clone https://github.com/tensorflow/serving.git tensorflow-serving\n$ cd tensorflow-serving/\n$ TESTDATA=\"$(pwd)/tensorflow_serving/servables/tensorflow/testdata\"\n$ TESTMODEL=\"$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two\"\n$ docker run -t --rm -p 8500:8500 -p 8501:8501 -v \"$TESTMODEL\" -e MODEL_NAME=half_plus_two tensorflow/serving\n```\n\n\n## Integration with Kubernetes (exec probe)\n\nKubernetes runs `exec` probes by executing a command within the target container.  This means the probe binary needs to be bundled inside the TensorFlow Serving image.  Below is an example docker file and an example kubernetes probe config.\n\n\nSample Docker file:\n```\nFROM tensorflow/serving\n\nRUN apt-get update \\\n    \u0026\u0026 apt-get -y install wget\n\nRUN TFS_PROBE_VERSION=1.0.1 \\\n    \u0026\u0026 wget -qO /bin/tfs_model_status_probe https://github.com/codycollier/tfs-model-status-probe/releases/download/v${TFS_PROBE_VERSION}/tfs_model_status_probe_${TFS_PROBE_VERSION}_linux_amd64 \\\n    \u0026\u0026 chmod +x /bin/tfs_model_status_probe\n```\n\nMore recent kubernetes versions have a [startup probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-startup-probes), which can be helpful when serving large, slow to load models.\n\nSample kubernetes config (subset):\n```\nspec:\n  containers:\n  - name: server\n    startupProbe:\n      exec:\n        command: [\"/bin/tfs_model_status_probe\", \"-addr=:8500\", \"-model-name=half_plus_two\"]\n      failureThreshold: 30\n      periodSeconds: 10\n    livenessProbe:\n      exec:\n        command: [\"/bin/tfs_model_status_probe\", \"-addr=:8500\", \"-model-name=half_plus_two\"]\n      failureThreshold: 1\n      periodSeconds: 15\n```\n\n\n## References\n\n\n[1] TensorFlow Serving\n\n[TensorFlow Serving](https://github.com/tensorflow/serving) is a server designed to load a TensorFlow model and provide access to it via gRPC and/or HTTP.\n\n\n[2] gRPC Health Probe\n\nAs noted previously, the tfs model status probe is inspired by the [grpc_health_probe](https://github.com/grpc-ecosystem/grpc-health-probe). The github actions files, goreleaser config, and small portions of the README were derived and adapted from the grpc-health-probe project.\n\n\n[3] TFS Model Service\n\nThe [ModelService](https://github.com/tensorflow/serving/blob/master/tensorflow_serving/apis/model_service.proto) is a [gRPC](https://grpc.io/) service with a `GetModelStatus()` rpc which is built-in to TensorFlow Serving.  The TensorFlow proto required to call the rpc is sourced from TensorFlow Serving and TensorFlow core.  See `./tfproto/` for more details.\n\n    \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodycollier%2Ftfs-model-status-probe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcodycollier%2Ftfs-model-status-probe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcodycollier%2Ftfs-model-status-probe/lists"}