{"id":13439191,"url":"https://github.com/onnx/onnx-tensorrt","last_synced_at":"2025-04-29T14:32:39.760Z","repository":{"id":37899329,"uuid":"131630299","full_name":"onnx/onnx-tensorrt","owner":"onnx","description":"ONNX-TensorRT: TensorRT backend for ONNX","archived":false,"fork":false,"pushed_at":"2025-03-07T22:57:11.000Z","size":2623,"stargazers_count":3057,"open_issues_count":180,"forks_count":545,"subscribers_count":67,"default_branch":"10.9-GA","last_synced_at":"2025-04-10T16:42:06.835Z","etag":null,"topics":["deep-learning","nvidia","onnx"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onnx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"docs/contributing.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-04-30T18:11:09.000Z","updated_at":"2025-04-10T11:45:54.000Z","dependencies_parsed_at":"2024-01-16T02:01:55.907Z","dependency_job_id":"b303e0fb-e8e1-4611-9e82-57d407b3e227","html_url":"https://github.com/onnx/onnx-tensorrt","commit_stats":null,"previous_names":[],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onnx%2Fonnx-tensorrt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onnx%2Fonnx-tensorrt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onnx%2Fonnx-tensorrt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onnx%2Fonnx-tensorrt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onnx","download_url":"https://codeload.github.com/onnx/onnx-tensorrt/tar.gz/refs/heads/10.9-GA","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251519505,"owners_count":21602342,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","nvidia","onnx"],"created_at":"2024-07-31T03:01:11.854Z","updated_at":"2025-04-29T14:32:39.731Z","avatar_url":"https://github.com/onnx.png","language":"C++","funding_links":[],"categories":["C++","\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++"],"sub_categories":[],"readme":"\u003c!--- SPDX-License-Identifier: Apache-2.0 --\u003e\n\n# TensorRT Backend For ONNX\n\nParses ONNX models for execution with [TensorRT](https://developer.nvidia.com/tensorrt).\n\nSee also the [TensorRT documentation](https://docs.nvidia.com/deeplearning/tensorrt/).\n\nFor the list of recent changes, see the [changelog](docs/Changelog.md).\n\nFor a list of commonly seen issues and questions, see the [FAQ](docs/faq.md).\n\nFor business inquiries, please contact researchinquiries@nvidia.com\n\nFor press and other inquiries, please contact Hector Marinez at hmarinez@nvidia.com\n\n## Supported TensorRT Versions\n\nDevelopment on the this branch is for the latest version of [TensorRT 10.9](https://developer.nvidia.com/nvidia-tensorrt-download) with full-dimensions and dynamic shape support.\n\nFor previous versions of TensorRT, refer to their respective branches.\n\n## Supported Operators\n\nCurrent supported ONNX operators are found in the [operator support matrix](docs/operators.md).\n\n# Installation\n\n### Dependencies\n\n - [Protobuf \u003e= 3.0.x](https://github.com/google/protobuf/releases)\n - [TensorRT 10.9](https://developer.nvidia.com/tensorrt)\n - [TensorRT 10.9 open source libraries](https://github.com/NVIDIA/TensorRT/)\n\n### Building\n\nFor building within docker, we recommend using and setting up the docker containers as instructed in the main [TensorRT repository](https://github.com/NVIDIA/TensorRT#setting-up-the-build-environment) to build the onnx-tensorrt library.\n\nOnce you have cloned the repository, you can build the parser libraries and executables by running:\n\n    cd onnx-tensorrt\n    mkdir build \u0026\u0026 cd build\n    cmake .. -DTENSORRT_ROOT=\u003cpath_to_trt\u003e \u0026\u0026 make -j\n    # Ensure that you update your LD_LIBRARY_PATH to pick up the location of the newly built library:\n    export LD_LIBRARY_PATH=$PWD:$LD_LIBRARY_PATH\n\nNote that this project has a dependency on CUDA. By default the build will look in `/usr/local/cuda` for the CUDA toolkit installation. If your CUDA path is different, overwrite the default path by providing `-DCUDA_TOOLKIT_ROOT_DIR=\u003cpath_to_cuda_install\u003e` in the CMake command.\n\nTo build with `protobuf-lite` support, add `-DUSE_ONNX_LITE_PROTO=1` to the end of the `cmake` command.\n\n### InstanceNormalizaiton Performance\n\nThere are two implementations of InstanceNormalization that may perform differently depending on various parameters. By default, the parser will use the native TensorRT implementation of InstanceNorm. Users that want to benchmark using the plugin implementation of InstanceNorm can unset the parser flag `kNATIVE_INSTANCENORM` prior to parsing the model. Note that the plugin implementation cannot be used for building version compatible or hardware compatible engines, and attempting to do so will result in an error.\n\nC++ Example:\n\n    // Unset the kNATIVE_INSTANCENORM flag to use the plugin implementation.\n    parser-\u003eunsetFlag(nvonnxparser::OnnxParserFlag::kNATIVE_INSTANCENORM);\n\nPython Example:\n\n    // Unset the NATIVE_INSTANCENORM flag to use the plugin implementation.\n    parser.clear_flag(trt.OnnxParserFlag.NATIVE_INSTANCENORM)\n\n## Executable Usage\n\nThere are currently two officially supported tools for users to quickly check if an ONNX model can parse and build into a TensorRT engine from an ONNX file.\n\nFor C++ users, there is the [trtexec](https://github.com/NVIDIA/TensorRT/tree/main/samples/opensource/trtexec) binary that is typically found in the `\u003ctensorrt_root_dir\u003e/bin` directory. The basic command of running an ONNX model is:\n\n`trtexec --onnx=model.onnx`\n\nRefer to the link or run `trtexec -h` for more information on CLI options.\n\nFor Python users, there is the [polygraphy](https://github.com/NVIDIA/TensorRT/tree/main/tools/Polygraphy) tool. The basic command for running an onnx model is:\n\n`polygraphy run model.onnx --trt`\n\nRefer to the link or run `polygraphy run -h` for more information on CLI options.\n\n### Python Modules\n\nPython bindings for the ONNX-TensorRT parser are packaged in the shipped `.whl` files.\n\nTensorRT 10.9 supports ONNX release 1.17.0. Install it with:\n\n    python3 -m pip install onnx==1.17.0\n\nThe ONNX-TensorRT backend can be installed by running:\n\n    python3 setup.py install\n\n## ONNX-TensorRT Python Backend Usage\n\nThe TensorRT backend for ONNX can be used in Python as follows:\n\n```python\nimport onnx\nimport onnx_tensorrt.backend as backend\nimport numpy as np\n\nmodel = onnx.load(\"/path/to/model.onnx\")\nengine = backend.prepare(model, device='CUDA:1')\ninput_data = np.random.random(size=(32, 3, 224, 224)).astype(np.float32)\noutput_data = engine.run(input_data)[0]\nprint(output_data)\nprint(output_data.shape)\n```\n\n## C++ Library Usage\n\nThe model parser library, libnvonnxparser.so, has its C++ API declared in this header:\n\n    NvOnnxParser.h\n\n### Tests\n\nAfter installation (or inside the Docker container), ONNX backend tests can be run as follows:\n\nReal model tests only:\n\n    python onnx_backend_test.py OnnxBackendRealModelTest\n\nAll tests:\n\n    python onnx_backend_test.py\n\nYou can use `-v` flag to make output more verbose.\n\n## Pre-trained Models\n\nPre-trained models in ONNX format can be found at the [ONNX Model Zoo](https://github.com/onnx/models)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonnx%2Fonnx-tensorrt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonnx%2Fonnx-tensorrt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonnx%2Fonnx-tensorrt/lists"}