{"id":23448309,"url":"https://github.com/triton-inference-server/client","last_synced_at":"2025-05-14T23:06:23.789Z","repository":{"id":37055360,"uuid":"352816666","full_name":"triton-inference-server/client","owner":"triton-inference-server","description":"Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.","archived":false,"fork":false,"pushed_at":"2025-04-18T19:32:01.000Z","size":4970,"stargazers_count":617,"open_issues_count":71,"forks_count":237,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-19T07:49:38.757Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/triton-inference-server.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-03-29T23:57:47.000Z","updated_at":"2025-04-18T04:06:26.000Z","dependencies_parsed_at":"2024-02-01T19:39:54.818Z","dependency_job_id":"09254ea6-8f3a-4a4c-a800-b3d65eb9cf47","html_url":"https://github.com/triton-inference-server/client","commit_stats":{"total_commits":636,"total_committers":50,"mean_commits":12.72,"dds":0.7405660377358491,"last_synced_commit":"ca9d47aae4fd5ebe40fb4ae68b837aadeb29e4d6"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fclient","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fclient/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fclient/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fclient/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/triton-inference-server","download_url":"https://codeload.github.com/triton-inference-server/client/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254243360,"owners_count":22038046,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-23T22:14:56.952Z","updated_at":"2025-05-14T23:06:18.779Z","avatar_url":"https://github.com/triton-inference-server.png","language":"Python","funding_links":[],"categories":["人工智能"],"sub_categories":[],"readme":"\u003c!--\n# Copyright 2021-2023, NVIDIA CORPORATION \u0026 AFFILIATES. All rights reserved.\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions\n# are met:\n#  * Redistributions of source code must retain the above copyright\n#    notice, this list of conditions and the following disclaimer.\n#  * Redistributions in binary form must reproduce the above copyright\n#    notice, this list of conditions and the following disclaimer in the\n#    documentation and/or other materials provided with the distribution.\n#  * Neither the name of NVIDIA CORPORATION nor the names of its\n#    contributors may be used to endorse or promote products derived\n#    from this software without specific prior written permission.\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY\n# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY\n# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n--\u003e\n\n[![License](https://img.shields.io/badge/License-BSD3-lightgrey.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\n# Triton Client Libraries and Examples\n\nTo simplify communication with Triton, the Triton project provides\nseveral client libraries and examples of how to use those\nlibraries. Ask questions or report problems in the main Triton [issues\npage](https://github.com/triton-inference-server/server/issues).\n\nThe provided client libraries are:\n\n* [C++ and Python APIs](#client-library-apis) that make it easy to\n  communicate with Triton from your C++ or Python application. Using\n  these libraries you can send either HTTP/REST or GRPC requests to\n  Triton to access all its capabilities: inferencing, status and\n  health, statistics and metrics, model repository management,\n  etc. These libraries also support using system and CUDA shared\n  memory for passing inputs to and receiving outputs from Triton.\n\n* [Java API](#client-library-apis) (contributed by Alibaba Cloud PAI Team)\n  that makes it easy to communicate with Triton from your Java application\n  using HTTP/REST requests. For now, only a limited feature subset is supported.\n\n* The [protoc\n  compiler](https://developers.google.com/protocol-buffers/docs/tutorials)\n  can generate a GRPC API in a large number of programming\n  languages.\n    * See [src/grpc_generated/go](src/grpc_generated/go) for an example for the\n    [Go programming language](https://golang.org/).\n    * See [src/grpc_generated/java](src/grpc_generated/java) for an example for\n    the Java and Scala programming languages.\n    * See [src/grpc_generated/javascript](src/grpc_generated/javascript) for\n    an example with JavaScript programming language.\n\nThere are also many example applications that show how to use these\nlibraries. Many of these examples use models from the [example model\nrepository](https://github.com/triton-inference-server/server/blob/main/docs/getting_started/quickstart.md#create-a-model-repository).\n\n* C++ and Python versions of *image_client*, an example application\n  that uses the C++ or Python client library to execute image\n  classification models on Triton. See [Image Classification\n  Example](#image-classification-example).\n\n* Several simple [C++ examples](src/c%2B%2B/examples) show\n  how to use the C++ library to communicate with Triton to perform\n  inferencing and other task. The C++ examples demonstrating the\n  HTTP/REST client are named with a *simple_http_* prefix and the\n  examples demonstrating the GRPC client are named with a\n  *simple_grpc_* prefix. See [Simple Example\n  Applications](#simple-example-applications).\n\n* Several simple [Python examples](src/python/examples)\n  show how to use the Python library to communicate with Triton to\n  perform inferencing and other task. The Python examples\n  demonstrating the HTTP/REST client are named with a *simple_http_*\n  prefix and the examples demonstrating the GRPC client are named with\n  a *simple_grpc_* prefix. See [Simple Example\n  Applications](#simple-example-applications).\n\n* Several simple [Java\n  examples](src/java/src/main/java/triton/client/examples) show how to\n  use the Java API to communicate with Triton to perform inferencing\n  and other task.\n\n* A couple of [Python examples that communicate with Triton using a\n  Python GRPC API](src/python/examples) generated by the\n  [protoc compiler](https://grpc.io/docs/guides/). *grpc_client.py* is\n  a simple example that shows simple API\n  usage. *grpc_image_client.py* is functionally equivalent to\n  *image_client* but that uses a generated GRPC client stub to\n  communicate with Triton.\n\n## Getting the Client Libraries And Examples\n\nThe easiest way to get the Python client library is to [use pip to\ninstall the tritonclient\nmodule](#download-using-python-package-installer-pip). You can also\ndownload the C++, Python and Java client libraries from [Triton GitHub\nrelease](#download-from-github), or [download a pre-built Docker image\ncontaining the client libraries](#download-docker-image-from-ngc) from\n[NVIDIA GPU Cloud (NGC)](https://ngc.nvidia.com).\n\nIt is also possible to build the client libraries with\n[cmake](#build-using-cmake).\n\n### Download Using Python Package Installer (pip)\n\nThe GRPC and HTTP client libraries are available as a Python package\nthat can be installed using a recent version of pip.\n\n```\n$ pip install tritonclient[all]\n```\n\nUsing *all* installs both the HTTP/REST and GRPC client\nlibraries. There are two optional packages available, *grpc* and\n*http* that can be used to install support specifically for the\nprotocol. For example, to install only the HTTP/REST client library\nuse,\n\n```\n$ pip install tritonclient[http]\n```\n\nThere is another optional package namely *cuda*, that must be installed\nin order to use cuda_shared_memory utilities. *all* specification will\ninstall the *cuda* package by default but in other cases *cuda* needs to\nbe explicitly specified for installing client with cuda_shared_memory\nsupport.\n\n```\n$ pip install tritonclient[http, cuda]\n```\n\nThe components of the install packages are:\n\n* http\n* grpc [ `service_pb2`, `service_pb2_grpc`, `model_config_pb2` ]\n* utils [ linux distribution will include `shared_memory` and `cuda_shared_memory`]\n\n### Download From GitHub\n\nThe client libraries can be downloaded from the [Triton GitHub release\npage](https://github.com/triton-inference-server/server/releases)\ncorresponding to the release you are interested in. The client\nlibraries are found in the \"Assets\" section of the release page in a\ntar file named after the version of the release and the OS, for\nexample, v2.3.0_ubuntu2004.clients.tar.gz.\n\nThe pre-built libraries can be used on the corresponding host system\nor you can install them into the Triton container to have both the\nclients and server in the same container.\n\n```bash\n$ mkdir clients\n$ cd clients\n$ wget https://github.com/triton-inference-server/server/releases/download/\u003ctarfile_path\u003e\n$ tar xzf \u003ctarfile_name\u003e\n```\n\nAfter installing, the libraries can be found in lib/, the headers in\ninclude/, the Python wheel files in python/, and the jar files in\njava/.  The bin/ and python/ directories contain the built examples\nthat you can learn more about below.\n\n### Download Docker Image From NGC\n\nA Docker image containing the client libraries and examples is\navailable from [NVIDIA GPU Cloud\n(NGC)](https://ngc.nvidia.com). Before attempting to pull the\ncontainer ensure you have access to NGC.  For step-by-step\ninstructions, see the [NGC Getting Started\nGuide](http://docs.nvidia.com/ngc/ngc-getting-started-guide/index.html).\n\nUse docker pull to get the client libraries and examples container\nfrom NGC.\n\n```bash\n$ docker pull nvcr.io/nvidia/tritonserver:\u003cxx.yy\u003e-py3-sdk\n```\n\nWhere \\\u003cxx.yy\\\u003e is the version that you want to pull. Within the\ncontainer the client libraries are in /workspace/install/lib, the\ncorresponding headers in /workspace/install/include, and the Python\nwheel files in /workspace/install/python. The image will also contain\nthe built client examples.\n\n**Important Note:** When running either the server or the client using\nDocker containers and using the\n[CUDA shared memory feature](https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_shared_memory.md#cuda-shared-memory)\nyou need to add `--pid host` flag when launching the containers. The reason is\nthat CUDA IPC APIs require the PID of the source and destination of the exported\npointer to be different. Otherwise, Docker enables PID namespace which may\nresult in equality between the source and destination PIDs. The error will be\nalways observed when both of the containers are started in the non-interactive\nmode.\n\n### Build Using CMake\n\nThe client library build is performed using CMake. To build the client\nlibraries and examples with all features, first change directory to\nthe root of this repo and checkout the release version of the branch\nthat you want to build (or the *main* branch if you want to build the\nunder-development version).\n\n```bash\n$ git checkout main\n```\n\nIf building the Java client you must first install Maven and a JDK\nappropriate for your OS. For example, for Ubuntu you should install\nthe `default-jdk` package:\n\n```\n$ apt-get install default-jdk maven\n```\n\nBuilding on Windows vs. non-Windows requires different invocations\nbecause Triton on Windows does not yet support all the build options.\n\n#### Non-Windows\n\nUse *cmake* to configure the build. You should adjust the flags depending on\nthe components of Triton Client you are working and would like to build.\n\nIf you are building on a release branch (or on a development branch\nthat is based off of a release branch), then you must also use\nadditional cmake arguments to point to that release branch for repos\nthat the client build depends on. For example, if you are building the\nr21.10 client branch then you need to use the following additional\ncmake flags:\n\n```\n-DTRITON_COMMON_REPO_TAG=r21.10\n-DTRITON_THIRD_PARTY_REPO_TAG=r21.10\n-DTRITON_CORE_REPO_TAG=r21.10\n```\n\nThen use *make* to build the clients and examples.\n\n```\n$ make cc-clients python-clients java-clients\n```\n\nWhen the build completes the libraries and examples can be found in\nthe install directory.\n\n#### Windows\n\nTo build the clients you must install an appropriate C++ compiler and\nother dependencies required for the build. The easiest way to do this\nis to create the [Windows min Docker\nimage](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/build.md#windows-10-min-image)\nand the perform the build within a container launched from that image.\n\n```\n\u003e docker run  -it --rm win10-py3-min powershell\n```\n\nIt is not necessary to use Docker or the win10-py3-min container for\nthe build, but if you do not you must install the appropriate\ndependencies onto your host system.\n\nNext use *cmake* to configure the build. If you are not building\nwithin the win10-py3-min container then you will likely need to adjust\nthe CMAKE_TOOLCHAIN_FILE location in the following command.\n\n```\n$ mkdir build\n$ cd build\n$ cmake -DVCPKG_TARGET_TRIPLET=x64-windows -DCMAKE_TOOLCHAIN_FILE='/vcpkg/scripts/buildsystems/vcpkg.cmake' -DCMAKE_INSTALL_PREFIX=install -DTRITON_ENABLE_CC_GRPC=ON -DTRITON_ENABLE_PYTHON_GRPC=ON -DTRITON_ENABLE_GPU=OFF -DTRITON_ENABLE_EXAMPLES=ON -DTRITON_ENABLE_TESTS=ON ..\n```\n\nIf you are building on a release branch (or on a development branch\nthat is based off of a release branch), then you must also use\nadditional cmake arguments to point to that release branch for repos\nthat the client build depends on. For example, if you are building the\nr21.10 client branch then you need to use the following additional\ncmake flags:\n\n```\n-DTRITON_COMMON_REPO_TAG=r21.10\n-DTRITON_THIRD_PARTY_REPO_TAG=r21.10\n-DTRITON_CORE_REPO_TAG=r21.10\n```\n\nThen use msbuild.exe to build.\n\n```\n$ msbuild.exe cc-clients.vcxproj -p:Configuration=Release -clp:ErrorsOnly\n$ msbuild.exe python-clients.vcxproj -p:Configuration=Release -clp:ErrorsOnly\n```\n\nWhen the build completes the libraries and examples can be found in\nthe install directory.\n\n## Client Library APIs\n\nThe C++ client API exposes a class-based interface. The commented\ninterface is available in\n[grpc_client.h](src/c%2B%2B/library/grpc_client.h),\n[http_client.h](src/c%2B%2B/library/http_client.h),\n[common.h](src/c%2B%2B/library/common.h).\n\nThe Python client API provides similar capabilities as the C++\nAPI. The commented interface is available in\n[grpc](src/python/library/tritonclient/grpc/__init__.py)\nand\n[http](src/python/library/tritonclient/http/__init__.py).\n\nThe Java client API provides similar capabilities as the Python API\nwith similar classes and methods.  For more information please refer\nto the [Java client directory](src/java).\n\n### HTTP Options\n\n#### SSL/TLS\n\nThe client library allows communication across a secured channel using HTTPS protocol. Just setting these SSL options do not ensure the secure communication. Triton server should be running behind `https://` proxy such as nginx. The client can then establish a secure channel to the proxy. The [`qa/L0_https`](https://github.com/triton-inference-server/server/blob/main/qa/L0_https/test.sh) in the server repository demonstrates how this can be achieved.\n\nFor C++ client, see `HttpSslOptions` struct that encapsulates these options in [http_client.h](src/c%2B%2B/library/http_client.h).\n\nFor Python client, look for the following options in [http/\\_\\_init\\_\\_.py](src/python/library/tritonclient/http/__init__.py):\n\n* ssl\n* ssl_options\n* ssl_context_factory\n* insecure\n\nThe [C++](src/c%2B%2B/examples/simple_http_infer_client.cc) and [Python](src/python/examples/simple_http_infer_client.py) examples\ndemonstrates how to use SSL/TLS settings on client side.\n\n\n#### Compression\n\nThe client library enables on-wire compression for HTTP transactions.\n\nFor C++ client, see `request_compression_algorithm` and `response_compression_algorithm` parameters in the `Infer` and `AsyncInfer` functions in [http_client.h](src/c%2B%2B/library/http_client.h). By default, the parameter is set as `CompressionType::NONE`.\n\nSimilarly, for Python client, see `request_compression_algorithm` and `response_compression_algorithm` parameters in `infer` and `async_infer` functions in [http/\\_\\_init\\_\\_.py](src/python/library/tritonclient/http/__init__.py).\n\nThe [C++](src/c%2B%2B/examples/simple_http_infer_client.cc) and [Python](src/python/examples/simple_http_infer_client.py) examples demonstrates how to use compression options.\n\n#### ORCA Header Metrics\n\nIn an effort to allow quick, on-demand metric retrieval for external load balancers such as the [Kubernetes Inference Gateway API](https://gateway-api-inference-extension.sigs.k8s.io/), Triton can include live KV-cache utilization and capacity metrics in the HTTP response header when processing inference requests. The motivation behind this method was to simplify the pipeline of metric scraping by not requiring separate service to hit the metrics endpoint, instead simply including a request header asking for metrics of a certain format in the response.\n\nTo use ORCA header metrics, Triton must be using the [TensorRT-LLM backend](https://github.com/triton-inference-server/tensorrtllm_backend) that exposes KV-cache metrics, and the HTTP inference request must include a header named `endpoint-load-metrics-format` with a value equal to one of the valid formats:\n\n`text`\n- Native HTTP, comma sepatared key-value pairs with the map fields elided into the top level scope by prepending the ‘\u003cmap_name\u003e’\n- Request header: `endpoint-load-metrics-format: text`\n- Ex. Response header: `endpoint-load-metrics: TEXT cpu_utilization=0.3, mem_utilization=0.8, rps_fractional=10.0, eps=1, named_metrics.custom_metric_util=0.4`\n\n`json`\n- JSON encoding of the metrics.\n- Request header: `endpoint-load-metrics-format: json`\n- Ex.  Response header:  `endpoint-load-metrics: JSON {“cpu_utilization”: 0.3, “mem_utilization”: 0.8, “rps_fractional”: 10.0, “eps”: 1, “named_metrics”: {“custom-metric-util”: 0.4}}`\n\n#### Python AsyncIO Support (Beta)\n\n*This feature is currently in beta and may be subject to change.*\n\nAdvanced users may call the Python client via `async` and `await` syntax. The\n[infer](src/python/examples/simple_http_aio_infer_client.py) example\ndemonstrates how to infer with AsyncIO.\n\nIf using SSL/TLS with AsyncIO, look for the `ssl` and `ssl_context` options in\n[http/aio/\\_\\_init\\_\\_.py](src/python/library/tritonclient/http/aio/__init__.py)\n\n#### Python Client Plugin API (Beta)\n\n*This feature is currently in beta and may be subject to change.*\n\n\nThe Triton Client Plugin API lets you register custom plugins to add or modify\nrequest headers. This is useful if you have gateway in front of Triton Server\nthat requires extra headers for each request, such as HTTP Authorization. By\nregistering the plugin, your gateway will work with Python clients without\nadditional configuration. Note that Triton Server does not implement\nauthentication or authorization mechanisms  and similarly,\nTriton Server is not the direct consumer of the additional headers.\n\nThe plugin must implement the `__call__` method. The signature\nof the `__call__` method should look like below:\n\n```python\nclass MyPlugin:\n  def __call__(self, request):\n       \"\"\"This method will be called for every HTTP request. Currently, the only\n       field that can be accessed by the request object is the `request.headers`\n       field. This field must be updated in-place.\n       \"\"\"\n       request.headers['my-header-key'] = 'my-header-value'\n```\n\nAfter the plugin implementation is complete, you can register the\nplugin by calling `register` on the `InferenceServerClient` object.\n\n```python\nfrom tritonclient.http import InferenceServerClient\n\nclient = InferenceServerClient(...)\n\n# Register the plugin\nmy_plugin = MyPlugin()\nclient.register_plugin(my_plugin)\n\n# All the method calls will update the headers according to the plugin\n# implementation.\nclient.infer(...)\n```\n\nTo unregister the plugin, you can call the `client.unregister_plugin()`\nfunction.\n\n##### Basic Auth\n\nYou can register the `BasicAuth` plugin that implements\n[Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).\n\n\n```python\nfrom tritonclient.grpc.auth import BasicAuth\nfrom tritonclient.grpc import InferenceServerClient\n\nbasic_auth = BasicAuth('username', 'password')\nclient = InferenceServerClient('...')\n\nclient.register_plugin(basic_auth)\n```\n\nThe example above shows how to register the plugin for\ngRPC client. The `BasicAuth` plugin can be registered\nsimilarly for HTTP and\n[AsyncIO](#python-asyncio-support-beta)\nclients.\n### GRPC Options\n\n#### SSL/TLS\n\nThe client library allows communication across a secured channel using gRPC protocol.\n\nFor C++ client, see `SslOptions` struct that encapsulates these options in [grpc_client.h](src/c%2B%2B/library/grpc_client.h).\n\nFor Python client, look for the following options in [grpc/\\_\\_init\\_\\_.py](src/python/library/tritonclient/grpc/__init__.py):\n\n* ssl\n* root_certificates\n* private_key\n* certificate_chain\n\nThe [C++](src/c%2B%2B/examples/simple_grpc_infer_client.cc) and [Python](src/python/examples/simple_grpc_infer_client.py) examples\ndemonstrates how to use SSL/TLS settings on client side. For information on the corresponding server-side parameters, refer to the\n[server documentation](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/inference_protocols.md#ssltls)\n\n#### Compression\n\nThe client library also exposes options to use on-wire compression for gRPC transactions.\n\nFor C++ client, see `compression_algorithm` parameter in the `Infer`, `AsyncInfer` and `StartStream` functions in [grpc_client.h](src/c%2B%2B/library/grpc_client.h). By default, the parameter is set as `GRPC_COMPRESS_NONE`.\n\nSimilarly, for Python client, see `compression_algorithm` parameter in `infer`, `async_infer` and `start_stream` functions in [grpc/\\_\\_init\\_\\_.py](src/python/library/tritonclient/grpc/__init__.py).\n\nThe [C++](src/c%2B%2B/examples/simple_grpc_infer_client.cc) and [Python](src/python/examples/simple_grpc_infer_client.py) examples demonstrates how to configure compression for clients. For information on the corresponding server-side parameters, refer to the [server documentation](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/inference_protocols.md#compression).\n\n#### GRPC KeepAlive\n\nTriton exposes GRPC KeepAlive parameters with the default values for both\nclient and server described [here](https://github.com/grpc/grpc/blob/master/doc/keepalive.md).\n\nYou can find a `KeepAliveOptions` struct/class that encapsulates these\nparameters in both the [C++](src/c%2B%2B/library/grpc_client.h) and\n[Python](src/python/library/tritonclient/grpc/__init__.py) client libraries.\n\nThere is also a [C++](src/c%2B%2B/examples/simple_grpc_keepalive_client.cc) and\n[Python](src/python/examples/simple_grpc_keepalive_client.py) example\ndemonstrating how to setup these parameters on the client-side. For information\non the corresponding server-side parameters, refer to the\n[server documentation](https://github.com/triton-inference-server/server/blob/main/docs/customization_guide/inference_protocols.md#grpc-keepalive)\n\n#### Custom GRPC Channel Arguments\n\nAdvanced users may require specific client-side GRPC Channel Arguments that are\nnot currently exposed by Triton through direct means. To support this, Triton\nallows users to pass custom channel arguments upon creating a GRPC client. When\nusing this option, it is up to the user to pass a valid combination of arguments\nfor their use case; Triton cannot feasibly test every possible combination of\nchannel arguments.\n\nThere is a [C++](src/c%2B%2B/examples/simple_grpc_custom_args_client.cc) and\n[Python](src/python/examples/simple_grpc_custom_args_client.py) example\ndemonstrating how to construct and pass these custom arguments upon creating\na GRPC client.\n\nYou can find a comprehensive list of possible GRPC Channel Arguments\n[here](https://grpc.github.io/grpc/core/group__grpc__arg__keys.html).\n\n#### Python AsyncIO Support (Beta)\n\n*This feature is currently in beta and may be subject to change.*\n\nAdvanced users may call the Python client via `async` and `await` syntax. The\n[infer](src/python/examples/simple_grpc_aio_infer_client.py) and\n[stream](src/python/examples/simple_grpc_aio_sequence_stream_infer_client.py)\nexamples demonstrate how to infer with AsyncIO.\n\n\n### Request Cancellation\n\nStarting from r23.10, triton python gRPC client can issue cancellation\nto inflight requests. This can be done by calling `cancel()` on the\nCallContext object returned by `async_infer()` API.\n\n```python\n  ctx = client.async_infer(...)\n  ctx.cancel()\n```\n\nFor streaming requests, `cancel_requests=True` can be sent to\n`stop_stream()` API to terminate all the inflight requests\nsent via this stream.\n\n```python\n  client.start_stream()\n  for _ in range(10):\n    client.async_stream_infer(...)\n\n  # Cancels all pending requests on stream closure rather than blocking until requests complete\n  client.stop_stream(cancel_requests=True)\n```\n\nSee more details about these APIs in\n[grpc/\\_client.py](src/python/library/tritonclient/grpc/_client.py).\n\nFor gRPC AsyncIO requests, an AsyncIO task wrapping an `infer()` coroutine can\nbe safely cancelled.\n\n```python\n  infer_task = asyncio.create_task(aio_client.infer(...))\n  infer_task.cancel()\n```\n\nFor gRPC AsyncIO streaming requests, `cancel()` can be called on the\nasynchronous iterator returned by `stream_infer()` API.\n\n```python\n  responses_iterator = aio_client.stream_infer(...)\n  responses_iterator.cancel()\n```\n\nSee more details about these APIs in\n[grpc/aio/\\__init__.py](src/python/library/tritonclient/grpc/aio/__init__.py).\n\nSee [request_cancellation](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/request_cancellation.md)\nin the server user-guide to learn about how this is handled on the\nserver side.\nIf writing your own gRPC clients in the language of choice consult\ngRPC guide on [cancellation](https://grpc.io/docs/guides/cancellation/#cancelling-an-rpc-call-on-the-client-side).\n\n### GRPC Status Codes\n\nStarting from release 24.08, Triton server introduces support for gRPC error\ncodes in streaming mode for all clients enhancing error reporting capabilities. When\nthis feature is enabled, the Triton server will return standard gRPC error codes\nand subsequently close the stream after delivering the error. This feature is\noptional can be enabled by adding header with `triton_grpc_error` key and `true` as\nvalue. See [grpc error\ncodes](https://github.com/triton-inference-server/server/tree/main#GRPC-Status-Codes) in the server to learn about how this is handled on the server side. See gRPC\nguide on [status-codes](https://grpc.io/docs/guides/status-codes/) for more details.\nBelow is a Python snippet to enable the feature. Without this header Triton server\nwill continue streaming in default mode returning error message and status inside\n`InferenceServerException` object within the callback provided.\n\n```python\n  triton_client = grpcclient.InferenceServerClient(triton_server_url)\n  # New added header key value\n  metadata = {\"triton_grpc_error\": \"true\"}\n  triton_client.start_stream(\n    callback=partial(callback, user_data), headers=metadata\n  )\n```\n## Simple Example Applications\n\nThis section describes several of the simple example applications and\nthe features that they illustrate.\n\n### Bytes/String Datatype\n\nSome frameworks support tensors where each element in the tensor is\nvariable-length binary data. Each element can hold a string or an\narbitrary sequence of bytes. On the client this datatype is BYTES (see\n[Datatypes](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#datatypes)\nfor information on supported datatypes).\n\nThe Python client library uses numpy to represent input and output\ntensors. For BYTES tensors the dtype of the numpy array should be\n'np.object_' as shown in the examples. For backwards compatibility\nwith previous versions of the client library, 'np.bytes_' can also be\nused for BYTES tensors. However, using 'np.bytes_' is not recommended\nbecause using this dtype will cause numpy to remove all trailing zeros\nfrom each array element. As a result, binary sequences ending in\nzero(s) will not be represented correctly.\n\nBYTES tensors are demonstrated in the C++ example applications\nsimple_http_string_infer_client.cc and\nsimple_grpc_string_infer_client.cc.  String tensors are demonstrated\nin the Python example application simple_http_string_infer_client.py\nand simple_grpc_string_infer_client.py.\n\n### System Shared Memory\n\nUsing system shared memory to communicate tensors between the client\nlibrary and Triton can significantly improve performance in some\ncases.\n\nUsing system shared memory is demonstrated in the C++ example\napplications simple_http_shm_client.cc and simple_grpc_shm_client.cc.\nUsing system shared memory is demonstrated in the Python example\napplication simple_http_shm_client.py and simple_grpc_shm_client.py.\n\nPython does not have a standard way of allocating and accessing shared\nmemory so as an example a simple [system shared memory\nmodule](src/python/library/tritonclient/utils/shared_memory)\nis provided that can be used with the Python client library to create,\nset and destroy system shared memory.\n\n### CUDA Shared Memory\n\nUsing CUDA shared memory to communicate tensors between the client\nlibrary and Triton can significantly improve performance in some\ncases.\n\nUsing CUDA shared memory is demonstrated in the C++ example\napplications simple_http_cudashm_client.cc and\nsimple_grpc_cudashm_client.cc.  Using CUDA shared memory is\ndemonstrated in the Python example application\nsimple_http_cudashm_client.py and simple_grpc_cudashm_client.py.\n\nPython does not have a standard way of allocating and accessing shared\nmemory so as an example a simple [CUDA shared memory\nmodule](src/python/library/tritonclient/utils/cuda_shared_memory)\nis provided that can be used with the Python client library to create,\nset and destroy CUDA shared memory. The module currently supports\nnumpy arrays ([example usage](src/python/examples/simple_http_cudashm_client.py))\nand DLPack tensors ([example usage](src/python/library/tests/test_dlpack.py)).\n\n### Client API for Stateful Models\n\nWhen performing inference using a [stateful\nmodel](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/architecture.md#stateful-models),\na client must identify which inference requests belong to the same\nsequence and also when a sequence starts and ends.\n\nEach sequence is identified with a sequence ID that is provided when\nan inference request is made. It is up to the clients to create a\nunique sequence ID. For each sequence the first inference request\nshould be marked as the start of the sequence and the last inference\nrequests should be marked as the end of the sequence.\n\nThe use of sequence ID and start and end flags are demonstrated in the\nC++ example applications simple_grpc_sequence_stream_infer_client.cc.\nThe use of sequence ID and start and end flags are demonstrated in the\nPython example application simple_grpc_sequence_stream_infer_client.py.\n\n## Image Classification Example\n\nThe image classification example that uses the C++ client API is\navailable at\n[src/c++/examples/image_client.cc](src/c%2B%2B/examples/image_client.cc). The\nPython version of the image classification client is available at\n[src/python/examples/image_client.py](src/python/examples/image_client.py).\n\nTo use image_client (or image_client.py) you must first have a running\nTriton that is serving one or more image classification models. The\nimage_client application requires that the model have a single image\ninput and produce a single classification output. If you don't have a\nmodel repository with image classification models see\n[QuickStart](https://github.com/triton-inference-server/server/blob/main/docs/getting_started/quickstart.md)\nfor instructions on how to create one.\n\nOnce Triton is running you can use the image_client application to\nsend inference requests. You can specify a single image or a directory\nholding images. Here we send a request for the inception_graphdef\nmodel for an image from the\n[qa/images](https://github.com/triton-inference-server/server/tree/main/qa/images).\n\n```bash\n$ image_client -m inception_graphdef -s INCEPTION qa/images/mug.jpg\nRequest 0, batch size 1\nImage 'qa/images/mug.jpg':\n    0.754130 (505) = COFFEE MUG\n```\n\nThe Python version of the application accepts the same command-line\narguments.\n\n```bash\n$ python image_client.py -m inception_graphdef -s INCEPTION qa/images/mug.jpg\nRequest 0, batch size 1\nImage 'qa/images/mug.jpg':\n     0.826384 (505) = COFFEE MUG\n```\n\nThe image_client and image_client.py applications use the client\nlibraries to talk to Triton. By default image_client instructs the\nclient library to use HTTP/REST protocol, but you can use the GRPC\nprotocol by providing the -i flag. You must also use the -u flag to\npoint at the GRPC endpoint on Triton.\n\n```bash\n$ image_client -i grpc -u localhost:8001 -m inception_graphdef -s INCEPTION qa/images/mug.jpg\nRequest 0, batch size 1\nImage 'qa/images/mug.jpg':\n    0.754130 (505) = COFFEE MUG\n```\n\nBy default the client prints the most probable classification for the\nimage. Use the -c flag to see more classifications.\n\n```bash\n$ image_client -m inception_graphdef -s INCEPTION -c 3 qa/images/mug.jpg\nRequest 0, batch size 1\nImage 'qa/images/mug.jpg':\n    0.754130 (505) = COFFEE MUG\n    0.157077 (969) = CUP\n    0.002880 (968) = ESPRESSO\n```\n\nThe -b flag allows you to send a batch of images for inferencing.\nThe image_client application will form the batch from the image or\nimages that you specified. If the batch is bigger than the number of\nimages then image_client will just repeat the images to fill the\nbatch.\n\n```bash\n$ image_client -m inception_graphdef -s INCEPTION -c 3 -b 2 qa/images/mug.jpg\nRequest 0, batch size 2\nImage 'qa/images/mug.jpg':\n    0.754130 (505) = COFFEE MUG\n    0.157077 (969) = CUP\n    0.002880 (968) = ESPRESSO\nImage 'qa/images/mug.jpg':\n    0.754130 (505) = COFFEE MUG\n    0.157077 (969) = CUP\n    0.002880 (968) = ESPRESSO\n```\n\nProvide a directory instead of a single image to perform inferencing\non all images in the directory.\n\n```\n$ image_client -m inception_graphdef -s INCEPTION -c 3 -b 2 qa/images\nRequest 0, batch size 2\nImage '/opt/tritonserver/qa/images/car.jpg':\n    0.819196 (818) = SPORTS CAR\n    0.033457 (437) = BEACH WAGON\n    0.031232 (480) = CAR WHEEL\nImage '/opt/tritonserver/qa/images/mug.jpg':\n    0.754130 (505) = COFFEE MUG\n    0.157077 (969) = CUP\n    0.002880 (968) = ESPRESSO\nRequest 1, batch size 2\nImage '/opt/tritonserver/qa/images/vulture.jpeg':\n    0.977632 (24) = VULTURE\n    0.000613 (9) = HEN\n    0.000560 (137) = EUROPEAN GALLINULE\nImage '/opt/tritonserver/qa/images/car.jpg':\n    0.819196 (818) = SPORTS CAR\n    0.033457 (437) = BEACH WAGON\n    0.031232 (480) = CAR WHEEL\n```\n\nThe [grpc_image_client.py](src/python/examples/grpc_image_client.py)\napplication behaves the same as the image_client except that instead\nof using the client library it uses the GRPC generated library to\ncommunicate with Triton.\n\n## Ensemble Image Classification Example Application\n\nIn comparison to the image classification example above, this example\nuses an ensemble of an image-preprocessing model implemented as a\n[DALI\nbackend](https://github.com/triton-inference-server/dali_backend) and\na TensorFlow Inception model. The ensemble model allows you to send\nthe raw image binaries in the request and receive classification\nresults without preprocessing the images on the client.\n\nTo try this example you should follow the [DALI ensemble example\ninstructions](https://github.com/triton-inference-server/dali_backend/tree/main/docs/examples/inception_ensemble).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriton-inference-server%2Fclient","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftriton-inference-server%2Fclient","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriton-inference-server%2Fclient/lists"}