{"id":23448300,"url":"https://github.com/triton-inference-server/onnxruntime_backend","last_synced_at":"2025-04-13T17:32:56.439Z","repository":{"id":38327695,"uuid":"290372011","full_name":"triton-inference-server/onnxruntime_backend","owner":"triton-inference-server","description":"The Triton backend for the ONNX Runtime.","archived":false,"fork":false,"pushed_at":"2024-05-21T21:26:53.000Z","size":259,"stargazers_count":114,"open_issues_count":66,"forks_count":53,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-05-21T22:28:28.256Z","etag":null,"topics":["backend","inference","onnx-runtime","triton-inference-server"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/triton-inference-server.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-26T02:14:08.000Z","updated_at":"2024-05-21T13:57:36.000Z","dependencies_parsed_at":"2023-09-22T07:16:07.324Z","dependency_job_id":"a4d52a0b-ea8b-4a38-95ec-c26cec8fd526","html_url":"https://github.com/triton-inference-server/onnxruntime_backend","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fonnxruntime_backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fonnxruntime_backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fonnxruntime_backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fonnxruntime_backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/triton-inference-server","download_url":"https://codeload.github.com/triton-inference-server/onnxruntime_backend/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248752484,"owners_count":21156102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["backend","inference","onnx-runtime","triton-inference-server"],"created_at":"2024-12-23T22:14:54.161Z","updated_at":"2025-04-13T17:32:56.432Z","avatar_url":"https://github.com/triton-inference-server.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\n# Copyright (c) 2020-2025, NVIDIA CORPORATION. All rights reserved.\n#\n# Redistribution and use in source and binary forms, with or without\n# modification, are permitted provided that the following conditions\n# are met:\n#  * Redistributions of source code must retain the above copyright\n#    notice, this list of conditions and the following disclaimer.\n#  * Redistributions in binary form must reproduce the above copyright\n#    notice, this list of conditions and the following disclaimer in the\n#    documentation and/or other materials provided with the distribution.\n#  * Neither the name of NVIDIA CORPORATION nor the names of its\n#    contributors may be used to endorse or promote products derived\n#    from this software without specific prior written permission.\n#\n# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY\n# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n# PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR\n# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY\n# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n--\u003e\n\n[![License](https://img.shields.io/badge/License-BSD3-lightgrey.svg)](https://opensource.org/licenses/BSD-3-Clause)\n\n# ONNX Runtime Backend\n\nThe Triton backend for the [ONNX\nRuntime](https://github.com/microsoft/onnxruntime). You can learn more\nabout Triton backends in the [backend\nrepo](https://github.com/triton-inference-server/backend). Ask\nquestions or report problems on the [issues\npage](https://github.com/triton-inference-server/onnxruntime_backend/issues).\n\nUse a recent cmake to build and install in a local directory.\nTypically you will want to build an appropriate ONNX Runtime\nimplementation as part of the build. You do this by specifying a ONNX\nRuntime version and a Triton container version that you want to use\nwith the backend. You can find the combination of versions used in a\nparticular Triton release in the TRITON_VERSION_MAP at the top of\nbuild.py in the branch matching the Triton release you are interested\nin. For example, to build the ONNX Runtime backend for Triton 23.04,\nuse the versions from TRITON_VERSION_MAP in the [r23.04 branch of\nbuild.py](https://github.com/triton-inference-server/server/blob/r23.04/build.py#L73).\n\n```\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install -DTRITON_BUILD_ONNXRUNTIME_VERSION=1.14.1 -DTRITON_BUILD_CONTAINER_VERSION=23.04 ..\n$ make install\n```\n\nThe resulting install/backends/onnxruntime directory can be added to a\nTriton installation as /opt/tritonserver/backends/onnxruntime.\n\nThe following required Triton repositories will be pulled and used in\nthe build. By default the \"main\" branch/tag will be used for each repo\nbut the listed CMake argument can be used to override.\n\n* triton-inference-server/backend: -DTRITON_BACKEND_REPO_TAG=[tag]\n* triton-inference-server/core: -DTRITON_CORE_REPO_TAG=[tag]\n* triton-inference-server/common: -DTRITON_COMMON_REPO_TAG=[tag]\n\nYou can add TensorRT support to the ONNX Runtime backend by using\n-DTRITON_ENABLE_ONNXRUNTIME_TENSORRT=ON. You can add OpenVino support\nby using -DTRITON_ENABLE_ONNXRUNTIME_OPENVINO=ON\n-DTRITON_BUILD_ONNXRUNTIME_OPENVINO_VERSION=\\\u003cversion\\\u003e, where\n\\\u003cversion\\\u003e is the OpenVino version to use and should match the\nTRITON_VERSION_MAP entry as described above. So, to build with both\nTensorRT and OpenVino support:\n\n```\n$ mkdir build\n$ cd build\n$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install -DTRITON_BUILD_ONNXRUNTIME_VERSION=1.14.1 -DTRITON_BUILD_CONTAINER_VERSION=23.04 -DTRITON_ENABLE_ONNXRUNTIME_TENSORRT=ON -DTRITON_ENABLE_ONNXRUNTIME_OPENVINO=ON -DTRITON_BUILD_ONNXRUNTIME_OPENVINO_VERSION=2021.2.200 ..\n$ make install\n```\n\n\n## ONNX Runtime with TensorRT optimization\nTensorRT can be used in conjunction with an ONNX model to further optimize the\nperformance. To enable TensorRT optimization you must set the model configuration\nappropriately. There are several optimizations available for TensorRT, like\nselection of the compute precision and workspace size. The optimization\nparameters and their description are as follows.\n\n\n* `precision_mode`: The precision used for optimization. Allowed values are \"FP32\", \"FP16\" and \"INT8\". Default value is \"FP32\".\n* `max_workspace_size_bytes`: The maximum GPU memory the model can use temporarily during execution. Default value is 1GB.\n* `int8_calibration_table_name`: Specify INT8 calibration table name. Applicable when precision_mode==\"INT8\" and the models do not contain Q/DQ nodes. If calibration table is provided for model with Q/DQ nodes then ORT session creation will fail.\n* `int8_use_native_calibration_table`: Calibration table to use. Allowed values are 1 (use native TensorRT generated calibration table) and 0 (use ORT generated calibration table). Default is 0. **Note: Latest calibration table file needs to be copied to trt_engine_cache_path before inference. Calibration table is specific to models and calibration data sets. Whenever new calibration table is generated, old file in the path should be cleaned up or be replaced.\n* `trt_engine_cache_enable`: Enable engine caching.\n* `trt_engine_cache_path`: Specify engine cache path.\n\nTo explore the usage of more parameters, follow the mapping table below and\ncheck [ONNX Runtime doc](https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#execution-provider-options) for detail.\n\n\u003e Please link to the latest ONNX Runtime binaries in CMake or build from\n[main branch of ONNX Runtime](https://github.com/microsoft/onnxruntime/tree/main) to enable latest options.\n\n### Parameter mapping between ONNX Runtime and Triton ONNXRuntime Backend\n\n| Key in Triton model configuration | Value in Triton model config                        | Corresponding TensorRT EP option in ONNX Runtime | Type   |\n| --------------------------------- | --------------------------------------------------- | :----------------------------------------------- | :----- |\n| max_workspace_size_bytes          | e.g: \"4294967296\"                                   | trt_max_workspace_size                           | int    |\n| trt_max_partition_iterations      | e.g: \"1000\"                                         | trt_max_partition_iterations                     | int    |\n| trt_min_subgraph_size             | e.g: \"1\"                                            | trt_min_subgraph_size                            | int    |\n| precision_mode                    | \"FP16\"                                              | trt_fp16_enable                                  | bool   |\n| precision_mode                    | \"INT8\"                                              | trt_int8_enable                                  | bool   |\n| int8_calibration_table_name       |                                                     | trt_int8_calibration_table_name                  | string |\n| int8_use_native_calibration_table | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_int8_use_native_calibration_table            | bool   |\n| trt_dla_enable                    |                                                     | trt_dla_enable                                   | bool   |\n| trt_dla_core                      | e.g: \"0\"                                            | trt_dla_core                                     | int    |\n| trt_engine_cache_enable           | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_engine_cache_enable                          | bool   |\n| trt_engine_cache_path             |                                                     | trt_engine_cache_path                            | string |\n| trt_engine_cache_prefix           |                                                     | trt_engine_cache_prefix                          | string |\n| trt_dump_subgraphs                | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_dump_subgraphs                               | bool   |\n| trt_force_sequential_engine_build | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_force_sequential_engine_build                | bool   |\n| trt_context_memory_sharing_enable | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_context_memory_sharing_enable                | bool   |\n| trt_layer_norm_fp32_fallback      | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_layer_norm_fp32_fallback                     | bool   |\n| trt_timing_cache_enable           | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_timing_cache_enable                          | bool   |\n| trt_timing_cache_path             |                                                     | trt_timing_cache_path                            | string |\n| trt_force_timing_cache            | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_force_timing_cache                           | bool   |\n| trt_detailed_build_log            | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_detailed_build_log                           | bool   |\n| trt_build_heuristics_enable       | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_build_heuristics_enable                      | bool   |\n| trt_sparsity_enable               | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_sparsity_enable                              | bool   |\n| trt_builder_optimization_level    | e.g: \"3\"                                            | trt_builder_optimization_level                   | int    |\n| trt_auxiliary_streams             | e.g: \"-1\"                                           | trt_auxiliary_streams                            | int    |\n| trt_tactic_sources                | e.g: \"-CUDNN,+CUBLAS\";                              | trt_tactic_sources                               | string |\n| trt_extra_plugin_lib_paths        |                                                     | trt_extra_plugin_lib_paths                       | string |\n| trt_profile_min_shapes            | e.g: \"input1:dim1xdimd2...,input2:dim1xdim2...,...\" | trt_profile_min_shapes                           | string |\n| trt_profile_max_shapes            | e.g: \"input1:dim1xdimd2...,input2:dim1xdim2...,...\" | trt_profile_max_shapes                           | string |\n| trt_profile_opt_shapes            | e.g: \"input1:dim1xdimd2...,input2:dim1xdim2...,...\" | trt_profile_opt_shapes                           | string |\n| trt_cuda_graph_enable             | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_cuda_graph_enable                            | bool   |\n| trt_dump_ep_context_model         | e.g: \"1\" or \"true\", \"0\" or \"false\"                  | trt_dump_ep_context_model                        | bool   |\n| trt_ep_context_file_path          |                                                     | trt_ep_context_file_path                         | string |\n| trt_ep_context_embed_mode         | e.g: \"1\"                                            | trt_ep_context_embed_mode                        | int    |\n\nThe section of model config file specifying these parameters will look like:\n\n```\n.\n.\n.\noptimization { execution_accelerators {\n  gpu_execution_accelerator : [ {\n    name : \"tensorrt\"\n    parameters { key: \"precision_mode\" value: \"FP16\" }\n    parameters { key: \"max_workspace_size_bytes\" value: \"1073741824\" }}\n    parameters { key: \"trt_engine_cache_enable\" value: \"1\" }}\n  ]\n}}\n.\n.\n.\n```\n\n## ONNX Runtime with CUDA Execution Provider optimization\nWhen GPU is enabled for ORT, CUDA execution provider is enabled. If TensorRT is\nalso enabled then CUDA EP is treated as a fallback option (only comes into\npicture for nodes which TensorRT cannot execute). If TensorRT is not enabled\nthen CUDA EP is the primary EP which executes the models. ORT enabled\nconfiguring options for CUDA EP to further optimize based on the specific model\nand user scenarios. There are several optimizations available, please refer to\nthe [ONNX Runtime doc](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#cuda-execution-provider)\nfor more details. To enable CUDA EP optimization you must set the model\nconfiguration appropriately:\n\n```\noptimization { execution_accelerators {\n  gpu_execution_accelerator : [ {\n    name : \"cuda\"\n    parameters { key: \"cudnn_conv_use_max_workspace\" value: \"0\" }\n    parameters { key: \"use_ep_level_unified_stream\" value: \"1\" }}\n  ]\n}}\n```\n\n### Deprecated Parameters\nThe way to specify these specific parameters as shown below is deprecated. For\nbackward compatibility, these parameters are still supported. Please use the\nabove method to specify the parameters.\n\n* `cudnn_conv_algo_search`: CUDA Convolution algorithm search configuration.\nAvailable options are 0 - EXHAUSTIVE (expensive exhaustive benchmarking using\ncudnnFindConvolutionForwardAlgorithmEx). This is also the default option,\n1 - HEURISTIC (lightweight heuristic based search using\ncudnnGetConvolutionForwardAlgorithm_v7), 2 - DEFAULT (default algorithm using\nCUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM)\n\n* `gpu_mem_limit`: CUDA memory limit. To use all possible memory pass in maximum\nsize_t. Defaults to SIZE_MAX.\n\n* `arena_extend_strategy`: Strategy used to grow the memory arena. Available\noptions are: 0 = kNextPowerOfTwo, 1 = kSameAsRequested. Defaults to 0.\n\n* `do_copy_in_default_stream`: Flag indicating if copying needs to take place on\nthe same stream as the compute stream in the CUDA EP. Available options are:\n0 = Use separate streams for copying and compute, 1 = Use the same stream for\ncopying and compute. Defaults to 1.\n\nIn the model config file, specifying these parameters will look like:\n\n```\n.\n.\n.\nparameters { key: \"cudnn_conv_algo_search\" value: { string_value: \"0\" } }\nparameters { key: \"gpu_mem_limit\" value: { string_value: \"4294967200\" } }\n.\n.\n.\n\n```\n\n\n## ONNX Runtime with OpenVINO optimization\n\n[OpenVINO](https://docs.openvinotoolkit.org/latest/index.html) can be\nused in conjunction with an ONNX model to further optimize\nperformance. To enable OpenVINO optimization you must set the model\nconfiguration as shown below.\n\n```\n.\n.\n.\noptimization { execution_accelerators {\n  cpu_execution_accelerator : [ {\n    name : \"openvino\"\n  } ]\n}}\n.\n.\n.\n```\n\n## Other Optimization Options with ONNX Runtime\n\nDetails regarding when to use these options and what to expect from them can be\nfound [here](https://onnxruntime.ai/docs/performance/tune-performance.html)\n\n### Model Config Options\n* `intra_op_thread_count`: Sets the number of threads used to parallelize the\nexecution within nodes. A value of 0 means ORT will pick a default which is\nnumber of cores.\n* `inter_op_thread_count`: Sets the number of threads used to parallelize the\nexecution of the graph (across nodes). If sequential execution is enabled this\nvalue is ignored.\nA value of 0 means ORT will pick a default which is number of cores.\n* `execution_mode`: Controls whether operators in the graph are executed\nsequentially or in parallel. Usually when the model has many branches, setting\nthis option to 1 .i.e. \"parallel\" will give you better performance. Default is\n0 which is \"sequential execution.\"\n* `level`: Refers to the graph optimization level. By default all optimizations\nare enabled. Allowed values are -1, 1 and 2. -1 refers to BASIC optimizations,\n1 refers to basic plus extended optimizations like fusions and 2 refers to all\noptimizations being disabled. Please find the details\n[here](https://onnxruntime.ai/docs/performance/graph-optimizations.html).\n\n```\noptimization {\n  graph : {\n    level : 1\n}}\n\nparameters { key: \"intra_op_thread_count\" value: { string_value: \"0\" } }\nparameters { key: \"execution_mode\" value: { string_value: \"0\" } }\nparameters { key: \"inter_op_thread_count\" value: { string_value: \"0\" } }\n\n```\n* `enable_mem_arena`: Use 1 to enable the arena and 0 to disable. See\n[this](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a0bbd62df2b3c119636fba89192240593)\nfor more information.\n* `enable_mem_pattern`: Use 1 to enable memory pattern and 0 to disable.\nSee [this](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#ad13b711736956bf0565fea0f8d7a5d75)\nfor more information.\n* `memory.enable_memory_arena_shrinkage`:\nSee [this](https://github.com/microsoft/onnxruntime/blob/master/include/onnxruntime/core/session/onnxruntime_run_options_config_keys.h)\nfor more information.\n* `session.use_device_allocator_for_initializers`: Use \"1\" to enable using device allocator for allocating initialized tensor memory and \"0\" to disable. The default is \"0\". See [this](https://onnxruntime.ai/docs/get-started/with-c.html) for more information.\n\n### Command line options\n\n#### Thread Pools\n\nWhen intra and inter op threads is set to 0 or a value higher than 1, by default\nORT creates threadpool per session. This may not be ideal in every scenario,\ntherefore ORT also supports global threadpools. When global threadpools are\nenabled ORT creates 1 global threadpool which is shared by every session.\nUse the backend config to enable global threadpool. When global threadpool is\nenabled, intra and inter op num threads config should also be provided via\nbackend config. Config values provided in model config will be ignored.\n\n```\n--backend-config=onnxruntime,enable-global-threadpool=\u003c0,1\u003e, --backend-config=onnxruntime,intra_op_thread_count=\u003cint\u003e , --backend-config=onnxruntime,inter_op_thread_count=\u003cint\u003e\n```\n\n#### Default Max Batch Size\n\nThe default-max-batch-size value is used for max_batch_size during\n[Autocomplete](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#auto-generated-model-configuration)\nwhen no other value is found. Assuming server was not launched with\n`--disable-auto-complete-config` command-line option, the onnxruntime backend\nwill set the max_batch_size of the model to this default value under the\nfollowing conditions:\n\n1. Autocomplete has determined the model is capable of batching requests.\n2. max_batch_size is 0 in the model configuration or max_batch_size\n   is omitted from the model configuration.\n\nIf max_batch_size \u003e 1 and no\n[scheduler](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#scheduling-and-batching)\nis provided, the dynamic batch scheduler will be used.\n\n```\n--backend-config=onnxruntime,default-max-batch-size=\u003cint\u003e\n```\n\nThe default value of `default-max-batch-size` is 4.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriton-inference-server%2Fonnxruntime_backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftriton-inference-server%2Fonnxruntime_backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriton-inference-server%2Fonnxruntime_backend/lists"}