{"id":23448294,"url":"https://github.com/triton-inference-server/model_analyzer","last_synced_at":"2025-05-14T18:02:43.118Z","repository":{"id":37059471,"uuid":"292107115","full_name":"triton-inference-server/model_analyzer","owner":"triton-inference-server","description":"Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.","archived":false,"fork":false,"pushed_at":"2025-04-21T16:28:57.000Z","size":6513,"stargazers_count":476,"open_issues_count":33,"forks_count":78,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-14T18:02:27.800Z","etag":null,"topics":["deep-learning","gpu","inference","performance-analysis"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/triton-inference-server.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-09-01T21:03:58.000Z","updated_at":"2025-05-09T08:45:32.000Z","dependencies_parsed_at":"2024-07-19T01:01:22.634Z","dependency_job_id":"ea6c9ae8-0004-4df8-bd9c-315dbafa0e7e","html_url":"https://github.com/triton-inference-server/model_analyzer","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fmodel_analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fmodel_analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fmodel_analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/triton-inference-server%2Fmodel_analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/triton-inference-server","download_url":"https://codeload.github.com/triton-inference-server/model_analyzer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254198452,"owners_count":22030964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gpu","inference","performance-analysis"],"created_at":"2024-12-23T22:14:53.735Z","updated_at":"2025-05-14T18:02:38.047Z","avatar_url":"https://github.com/triton-inference-server.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nCopyright (c) 2020-2023, NVIDIA CORPORATION \u0026 AFFILIATES. All rights reserved.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n--\u003e\n\n[![License](https://img.shields.io/badge/License-Apache_2.0-lightgrey.svg)](https://opensource.org/licenses/Apache-2.0)\n\n# Triton Model Analyzer\n\n\u003e [!Warning]\n\u003e\n\u003e ##### LATEST RELEASE\n\u003e\n\u003e You are currently on the `main` branch which tracks under-development progress towards the next release. \u003cbr\u003e\n\u003e The latest release of the Triton Model Analyzer is 1.47.0 and is available on branch\n\u003e [r24.12](https://github.com/triton-inference-server/model_analyzer/tree/r24.12).\n\nTriton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements.\n\u003cbr\u003e\u003cbr\u003e\n\n# Features\n\n### Search Modes\n\n- [Optuna Search](docs/config_search.md#optuna-search-mode) **_-ALPHA RELEASE-_** allows you to search for every parameter that can be specified in the model configuration, using a hyperparameter optimization framework. Please see the [Optuna](https://optuna.org/) website if you are interested in specific details on how the algorithm functions.\n\n- [Quick Search](docs/config_search.md#quick-search-mode) will **sparsely** search the [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size),\n  [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and\n  [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups) spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration\n\n- [Automatic Brute Search](docs/config_search.md#automatic-brute-search) will **exhaustively** search the\n  [Max Batch Size](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#maximum-batch-size),\n  [Dynamic Batching](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#dynamic-batcher), and\n  [Instance Group](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_configuration.md#instance-groups)\n  parameters of your model configuration\n\n- [Manual Brute Search](docs/config_search.md#manual-brute-search) allows you to create manual sweeps for every parameter that can be specified in the model configuration\n\n### Model Types\n\n- [Ensemble](docs/model_types.md#ensemble): Model Analyzer can help you find the optimal\n  settings when profiling an ensemble model\n\n- [BLS](docs/model_types.md#bls): Model Analyzer can help you find the optimal\n  settings when profiling a BLS model\n\n- [Multi-Model](docs/model_types.md#multi-model): Model Analyzer can help you\n  find the optimal settings when profiling multiple concurrent models\n\n- [LLM](docs/model_types.md#llm): Model Analyzer can help you\n  find the optimal settings when profiling Large Language Models\n\n### Other Features\n\n- [Detailed and summary reports](docs/report.md): Model Analyzer is able to generate\n  summarized and detailed reports that can help you better understand the trade-offs\n  between different model configurations that can be used for your model.\n\n- [QoS Constraints](docs/config.md#constraint): Constraints can help you\n  filter out the Model Analyzer results based on your QoS requirements. For\n  example, you can specify a latency budget to filter out model configurations\n  that do not satisfy the specified latency threshold.\n  \u003cbr\u003e\u003cbr\u003e\n\n# Examples and Tutorials\n\n### **Single Model**\n\nSee the [Single Model Quick Start](docs/quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model.\n\n### **Multi Model**\n\nSee the [Multi-model Quick Start](docs/mm_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU.\n\n### **Ensemble Model**\n\nSee the [Ensemble Model Quick Start](docs/ensemble_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model.\n\n### **BLS Model**\n\nSee the [BLS Model Quick Start](docs/bls_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model.\n\u003cbr\u003e\u003cbr\u003e\n\n# Documentation\n\n- [Installation](docs/install.md)\n- [Model Analyzer CLI](docs/cli.md)\n- [Launch Modes](docs/launch_modes.md)\n- [Configuring Model Analyzer](docs/config.md)\n- [Model Analyzer Metrics](docs/metrics.md)\n- [Model Config Search](docs/config_search.md)\n- [Model Types](docs/model_types.md)\n- [Checkpointing](docs/checkpoints.md)\n- [Model Analyzer Reports](docs/report.md)\n- [Deployment with Kubernetes](docs/kubernetes_deploy.md)\n  \u003cbr\u003e\u003cbr\u003e\n\n# Terminology\n\nBelow are definitions of some commonly used terms in Model Analyzer:\n\n- **Model Type** - Category of model being profiled. Examples of this include single, multi, ensemble, BLS, etc..\n- **Search Mode** - How Model Analyzer explores the possible configuration space when profiling. This is either exhaustive (brute) or heuristic (quick/optuna).\n- **Model Config Search** - The cross product of model type and search mode.\n- **Launch Mode** - How the Triton Server is deployed and used by Model Analyzer.\n\n# Reporting problems, asking questions\n\nWe appreciate any feedback, questions or bug reporting regarding this\nproject. When help with code is needed, follow the process outlined in\nthe Stack Overflow (https://stackoverflow.com/help/mcve)\ndocument. Ensure posted examples are:\n\n- minimal – use as little code as possible that still produces the\n  same problem\n\n- complete – provide all parts needed to reproduce the problem. Check\n  if you can strip external dependency and still show the problem. The\n  less time we spend on reproducing problems the more time we have to\n  fix it\n\n- verifiable – test the code you're about to provide to make sure it\n  reproduces the problem. Remove all other problems that are not\n  related to your request/question.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriton-inference-server%2Fmodel_analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftriton-inference-server%2Fmodel_analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftriton-inference-server%2Fmodel_analyzer/lists"}