{"id":51366563,"url":"https://github.com/ai-dynamo/aitune","last_synced_at":"2026-07-03T02:08:07.809Z","repository":{"id":368049051,"uuid":"1166381425","full_name":"ai-dynamo/aitune","owner":"ai-dynamo","description":"NVIDIA AITune is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on NVIDIA GPUs.","archived":false,"fork":false,"pushed_at":"2026-06-03T06:42:16.000Z","size":9981,"stargazers_count":276,"open_issues_count":2,"forks_count":31,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-28T22:07:01.898Z","etag":null,"topics":["deep-learning","inference","nvidia","nvidia-gpu"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai-dynamo.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":"COPYRIGHT","agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-25T06:53:17.000Z","updated_at":"2026-06-27T22:14:10.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ai-dynamo/aitune","commit_stats":null,"previous_names":["ai-dynamo/aitune"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/ai-dynamo/aitune","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-dynamo%2Faitune","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-dynamo%2Faitune/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-dynamo%2Faitune/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-dynamo%2Faitune/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai-dynamo","download_url":"https://codeload.github.com/ai-dynamo/aitune/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai-dynamo%2Faitune/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35069263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-03T02:00:05.635Z","response_time":110,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","inference","nvidia","nvidia-gpu"],"created_at":"2026-07-03T02:08:07.148Z","updated_at":"2026-07-03T02:08:07.798Z","avatar_url":"https://github.com/ai-dynamo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\n# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION \u0026 AFFILIATES. All rights reserved.\n# SPDX-License-Identifier: Apache-2.0\n--\u003e\n\n# NVIDIA AITune\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)\n[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)\n[![PyTorch](https://img.shields.io/badge/PyTorch-2.7+-red.svg)](https://pytorch.org/)\n\n**NVIDIA AITune** is an inference toolkit designed for tuning and deploying Deep Learning models with a focus on NVIDIA GPUs. It provides model tuning capabilities through compilation and conversion paths that can significantly improve inference speed and efficiency across various AI workloads including Computer Vision, Natural Language Processing, Speech Recognition, and Generative AI.\n\nThe toolkit enables seamless tuning of PyTorch models and pipelines using various backends such as TensorRT, Torch-TensorRT, TorchAO, Torch Inductor, and ONNX Runtime through a single Python API. The resulting tuned models are ready for deployment in production environments.\n\nNVIDIA AITune works with your environment — relying first on your software versions — and selects the best-performing backend for your software and hardware setup, guiding you to supported technologies.\n\n**Note**: This is the first release. The API may change in future versions.\n\n   **NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those    retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses.  You are solely responsible for finding, reviewing and complying with all applicable terms, conditions, and licenses, and for verifying the security, integrity and suitability of any retrieved materials for your specific use case. This software is provided \"AS IS\", without warranty of any kind. The author makes no representations or warranties regarding any retrieved materials, and assumes no liability for any losses, damages, liabilities or legal consequences from your use or inability to use this software or any retrieved materials. Use this software and the retrieved materials at your own risk.**\n\n## Features at Glance\n\nThe distinct capabilities of NVIDIA AITune are summarized in the feature matrix:\n\n| Feature                     | Description                                                                                                               |\n|-----------------------------|---------------------------------------------------------------------------------------------------------------------------|\n| Ease-of-use                 | Single line of code to run all possible tuning paths directly from your source code                                       |\n| Wide Backend Support        | Compatible with various tuning backends including TensorRT, Torch-TensorRT, TorchAO, Torch Inductor, and ONNX Runtime    |\n| Model Tuning                | Enhance the performance of models such as ResNET and BERT for efficient inference deployment                              |\n| Pipeline Tuning             | Streamline Python code pipelines for models such as Stable Diffusion and Flux using seamless model wrapping and tuning    |\n| Model Export and Conversion | Automate the process of exporting and converting models between various formats with focus on TensorRT, Torch-TensorRT, and ONNX Runtime |\n| Correctness Testing         | Ensures tuned models produce correct outputs by validating on provided data samples                                       |\n| Performance Profiling       | Profiles models to select the optimal backend based on performance metrics such as latency and throughput                 |\n| Model Persistence           | Save and load tuned models for production deployment with flexible storage options                                        |\n| JIT tuning                  | Just-in-time tuning of a model or a pipeline without any code changes required                                            |\n\n## When to Use AITune\n\nAITune provides compute graph optimizations for PyTorch models at the `nn.Module` level. Use AITune when you want automated inference optimization with minimal code changes.\n\nIf your model is supported by a dedicated serving framework and benefits from runtime optimizations (e.g. continuous batching, speculative decoding), use frameworks like TensorRT-LLM, vLLM, or SGLang for best performance. Use AITune for general PyTorch models and pipelines that lack such specialized tooling.\n\n## Prerequisites\n\nBefore proceeding with the installation of NVIDIA AITune, ensure your system meets the following criteria:\n\n* **Operating System**: Linux (Ubuntu 22.04+ recommended)\n* **Python**: Version `3.10` or newer\n* **PyTorch**: Version `2.7` or newer\n* **TensorRT**: Version `10.5.0` or higher (for TensorRT backend)\n* **NVIDIA GPU**: Required for GPU-accelerated tuning\n\nYou can use NGC Containers for PyTorch which contain all necessary dependencies:\n\n* [PyTorch NGC Container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch)\n\n## Install\n\nNVIDIA AITune can be installed from `pypi.org`.\n\n### Installing from PyPI (Recommended)\n\n```bash\npip install --extra-index-url https://pypi.nvidia.com aitune\n```\n\n### Installing from Source\n\n```bash\n# Clone the repository\ngit clone https://github.com/ai-dynamo/aitune\ncd aitune\npip install --extra-index-url https://pypi.nvidia.com .\n```\n\nor use editable mode for development:\n\n```bash\npip install --extra-index-url https://pypi.nvidia.com -e .\n```\n\n## Quick Start\n\nThis quick start provides examples of tuning and deployment paths available in NVIDIA AITune.\n\nNVIDIA AITune enables seamless tuning of models for deployment (for example, converting them to TensorRT) without requiring changes to your original Python pipelines.\n\nNVIDIA AITune supports two modes:\n\n* Ahead-of-time tuning — provide a model or a pipeline, and a dataset/dataloader. You can either rely on `inspect` to detect promising modules to tune or manually select them.\n* Just-in-time tuning — set a special environment variable, run your script without changes, and AITune will, on the fly, detect modules and tune them one by one.\n\nAhead-of-time mode is more powerful and allows you to tweak more settings, whereas just-in-time works out of the box but offers less control over the tuning process. For a more detailed comparison, see the [Comparison between AOT and JIT tuning](#comparison-between-ahead-of-time-and-just-in-time-tuning) section.\n\n### Enabling logging\n\nThe tuning process guides the user through decisions and steps that are performed to tune every selected module.\n\nWe recommend to enable the INFO logging level for better verbosity.\n```python\nimport logging\n\nlogging.basicConfig(level=logging.INFO, force=True)\n```\n\n### Ahead-of-time tuning\n\nThe code below demonstrates Stable Diffusion pipeline tuning.\n\nYou can annotate `torch.nn.Module`s manually or use the `inspect` functionality to have modules picked automatically; you can then verify them and schedule them for tuning.\n\nFirst, install the required third-party dependencies:\n\n```bash\npip install transformers diffusers torch\n```\n\nThen initialize the pipeline:\n\n```python\n# HuggingFace dependencies\nimport torch\nfrom diffusers import DiffusionPipeline\n\n# Import AITune\nimport aitune.torch as ait\n\n# Initialize pipeline\npipe = DiffusionPipeline.from_pretrained(\"stable-diffusion-v1-5/stable-diffusion-v1-5\", torch_dtype=torch.float16)\npipe.to(\"cuda\")\n```\n\nNext, `inspect` the pipeline components and display the summary:\n\n```python\n# Prepare input data\ninput_data = [{\"prompt\": \"A beautiful landscape with mountains and a lake\"}]\n\n# Inspect pipeline to get modules\nmodules_info = ait.inspect(pipe, input_data)\n\n\n# Optional: inference function, if you need more control over execution\ndef infer(prompt):\n    return pipe(prompt, width=1024, height=1024, num_inference_steps=10)\n\n\n# modules_info = ait.inspect(pipe, input_data, inference_function=infer)\n\n# Display modules info\nmodules_info.describe()\n```\n\nFinally, `wrap` the selected modules and `tune` within the pipeline:\n\n```python\n# Wrap modules for tuning\nmodules = modules_info.get_modules()\npipe = ait.wrap(pipe, modules)\n\n# Tune pipeline\nait.tune(pipe, input_data)\n```\n\nAt this point, you can use the pipeline to generate predictions with the tuned models directly in Python:\n\n```python\n# Run inference on tuned pipeline\nimages = pipe([\"A beautiful landscape with mountains and a lake\"])\nimage = images[0][0]\n\n# Save image for preview\nimage.save(\"landscape.png\")\n```\n\nOnce the pipeline has been tuned, you can save the best-performing version of the modules for later deployment:\n\n```python\nait.save(pipe, \"tuned_pipe.ait\")\n```\n\nAnd load the tuned pipeline directly:\n\n```python\npipe = DiffusionPipeline.from_pretrained(\"stable-diffusion-v1-5/stable-diffusion-v1-5\", torch_dtype=torch.float16)\npipe.to(\"cuda\")\nait.load(pipe, \"tuned_pipe.ait\")\n```\n\n### Just-in-time tuning\n\nIn this mode, there is no need to modify the user's code. At the beginning, AITune uses a few inferences to detect model architecture and hierarchy of a model. Then it tries to tune modules one by one starting from the top. If there is one of the following conditions:\n\n* a graph break is detected, i.e., torch.nn.Module contains conditional logic on inputs, meaning there is no guarantee of a static, correct graph of computations, or\n* there is an error during tuning\n\nthat module is left unchanged and AITune tries to tune its children. This process continues until the module depth reaches a configured limit.\n\nFirst, install the required third-party dependencies:\n\n```bash\npip install transformers diffusers torch\n```\n\nPrepare the example script for tuning `my_script.py`:\n\n```python\n# Enable JIT tuning\nimport aitune.torch.jit.enable\n\n# HuggingFace dependencies\nimport torch\nfrom diffusers import DiffusionPipeline\n\n# Initialize pipeline\npipe = DiffusionPipeline.from_pretrained(\"stable-diffusion-v1-5/stable-diffusion-v1-5\", torch_dtype=torch.float16)\npipe.to(\"cuda\")\n\n# First call - tuning the model\npipe(\"A beautiful landscape with mountains and a lake\")\n\n# Second call - using tuned model\npipe(\"A beautiful landscape with mountains and a lake\")\n```\n\nYou can then run your script:\n\n```bash\npython my_script.py\n```\n\n*Note*: The `import aitune.torch.jit.enable` must be a first import in your code. The alternative option is to use `export AUTOWRAPT_BOOTSTRAP=aitune_enable_jit_tuning` to avoid any source code modification.\n\n#### Configuring just-in-time tuning\n\nIf there is a need to adjust just-in-time options, you can do it but currently this requires modifying code to import the JIT config:\n\n```python\nfrom aitune.torch.jit.config import config\nfrom aitune.torch.backend import TensorRTBackend\nfrom aitune.torch.tune_strategy import FirstWinsStrategy\n\nconfig.max_depth_level = 1  # change the default maximum depth level for nested modules to be tuned\nconfig.detect_graph_breaks = False  # turn off graph break detection\nconfig.strategy = FirstWinsStrategy(backends=[TensorRTBackend()])  # change the tune strategy\n```\n\n## Comparison between ahead-of-time and just-in-time tuning\n\nThe ahead-of-time tuning gives you the most control over the tuning process:\n\n* it detects the batch axis and dynamic axes (axes that change shape independently of batch size, e.g., sequence length in LLMs)\n* allows picking modules to tune\n* you can pick a tuning strategy (e.g., best throughput) for the whole process or per-module\n* you can pick tuning backends (e.g., TensorRT, TorchInductor, TorchAO, ONNXRuntime) which will be used by the strategy\n* you can mix different backends in the same model/pipeline\n* you can manually verify the tuning process (note: AITune performs basic checks for NaNs and errors)\n* you can save the resulting artifact and later read it from disk\n\nThe big advantage of just-in-time tuning is that you don't need to modify the user's script to tune a model. However, it has some disadvantages - since it cannot access data directly (you don't provide a dataloader):\n\n* it cannot deduce batch size nor do benchmarking\n* input/output shapes depend on the data seen, so for example, TRT backend will build a profile only for that data\n* it needs at least two inference calls - first to get model/pipeline hierarchy and second one for actual tuning\n* if you need dynamic axes (e.g., TRT backend), you need to provide two different batch sizes\n* there is limited support of strategies due to unknown batch size\n* you can specify backends for the whole model\n\nThe following table summarizes the difference between modes:\n\n| Feature                 | Ahead-of-time         | Just-in-time                  |\n|-------------------------|-----------------------|-------------------------------|\n| Detecting dynamic axes  | Yes                   | Yes                           |\n| Extrapolating batches   | Yes                   | No                            |\n| Benchmarking            | Yes                   | No (no extrapolating batches) |\n| Modules for tuning      | User has full control | Picked automatically          |\n| Selecting tune strategy | Global or per module  | Global                        |\n| Available strategies    | All                   | Limited (no benchmarking)     |\n| Tune time               | Slow                  | Quick                         |\n| Saving artifacts        | Yes                   | No                            |\n| Load tuned model time   | Quick                 | Re-tuning required            |\n| Code changes required   | Yes                   | No                            |\n| Caching                 | Yes                   | No                            |\n\nNote: Currently, JIT mode does not support caching results, i.e., every time a new Python interpreter starts, the tuning process starts from scratch.\n\n## Core Functionalities\n\n### Inspect for AOT tuning\n\nThe `inspect` function allows you to analyze PyTorch models and pipelines to understand their structure, parameters, and execution flow. It provides detailed insights into model architecture and helps identify tuning opportunities.\n\n```python\nimport aitune.torch as ait\nimport torch.nn as nn\n\n\nclass SimpleModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.linear = nn.Linear(100, 10)\n\n    def forward(self, x):\n        return self.linear(x)\n\n\nmodel = SimpleModel()\n\n# Inspect the model\nait.inspect(model, dataset)\n```\n\n### Inspect for JIT tuning\n\nJIT tuning also has a corresponding `inspect` mode which gathers information about the model/pipeline and allows checking model input and output arguments, hierarchy of the model, etc.\n\nHere is a short snippet how to use it:\n\n```python\n# required imports\nimport aitune.torch.jit.enable_inspection as inspection\n\n# your code goes here\n# ...\n\n# you can export report to html file\ninspection.save_report(\"filename.html\", \"YOUR_MODEL_NAME\")\n```\n\n### Tune\n\nThe `tune` function is the core functionality that automatically tunes your PyTorch models and pipelines for optimal inference performance. It supports various backends and automatically selects the best performing configuration.\n\n```python\nimport aitune.torch as ait\nimport torch\n\n# Define your model\nmodel = SimpleModel()\n\n# Wrap the model\nmodel = ait.Module(model)\n\n\n# Define inference function\ndef inference_fn(x):\n    return model(x)\n\n\n# Tune the model\nait.tune(\n    func=inference_fn,\n    dataset=torch.randn(1, 100),\n)\n```\n\n### Save\n\nThe `save` function allows you to persist tuned models for later use. It stores tuned and original module weights together in a single file with a `.ait` extension. Apart from the checkpoint file, there is also a SHA hash file.\n\n```python\n# Save the tuned model\nimport aitune.torch as ait\n\nait.save(model, \"tuned_model.ait\")\n```\n\nExample output:\n\n```bash\ncheckpoints/\n├── tuned_model\n├── tuned_model.ait\n└── tuned_model_sha256_sums.txt\n```\n\nYou can copy the checkpoint file `tuned_model.ait` and SHA sums file to a target host or folder to use it for inference.\n\n*Note:* We recommend to deploy `*.ait` package on the same hardware as tuning has been performed for functional and performance compatibility.\n\n### Load\n\nThe `load` function enables you to load previously tuned models from a checkpoint file.\n\n```python\n# Load the tuned model\nimport aitune.torch as ait\n\ntuned_model = ait.load(model, \"tuned_model.ait\")\n```\n\nOn first load, the checkpoint file is decompressed and the tuned and original module weights are loaded. Subsequent loads will use the decompressed weights from the same folder.\n\n## Backends\n\nNVIDIA AITune supports multiple tuning backends, each with different characteristics and use cases. The backends align with a common interface for the build and inference process.\n\n### TensorRT Backend\n\nThe TensorRT backend provides highly optimized inference using NVIDIA's TensorRT engine. It offers the best performance for production deployments. The backend integrates [TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) in a seamless flow.\n\n```python\nfrom aitune.torch.backend import TensorRTBackend, TensorRTBackendConfig, ONNXAutoCastConfig\n\nconfig = TensorRTBackendConfig(quantization_config=ONNXAutoCastConfig())  # FP16 autocast through ModelOpt\nbackend = TensorRTBackend(config)\n```\n\n#### CUDA Graphs Support\n\nThe TensorRT backend supports CUDA Graphs for reduced CPU overhead and improved inference performance. CUDA Graphs automatically capture and replay GPU operations, eliminating kernel launch overhead for repeated inference calls. This feature is disabled by default.\n\nKeep in mind that graphs are automatically recaptured when input shapes change.\n\n```python\nfrom aitune.torch.backend import TensorRTBackend, TensorRTBackendConfig\n\n# Enable CUDA Graphs for optimized inference\nconfig = TensorRTBackendConfig(use_cuda_graphs=True)\nbackend = TensorRTBackend(config)\n```\n\n### Torch-TensorRT Backend (JIT)\n\nTorch-TensorRT JIT backend integrates TensorRT tuning directly into PyTorch, providing seamless tuning without model conversion through\n`torch.compile`.\n\n```python\nimport torch\nfrom aitune.torch.backend import TorchTensorRTJitBackend, TorchTensorRTJitBackendConfig, TorchTensorRTConfig\n\nconfig = TorchTensorRTJitBackendConfig(compile_config=TorchTensorRTConfig(enabled_precisions={torch.float16}))\nbackend = TorchTensorRTJitBackend(config)\n```\n\n### Torch-TensorRT Backend (AOT)\n\nTorch-TensorRT backend integrates TensorRT tuning directly into PyTorch, providing seamless tuning without model conversion through `torch_tensorrt.compile`.\n\n```python\nimport torch\nfrom aitune.torch.backend import TorchTensorRTAotBackend, TorchTensorRTAotBackendConfig, TorchTensorRTConfig\n\nconfig = TorchTensorRTAotBackendConfig(compile_config=TorchTensorRTConfig(enabled_precisions={torch.float16}))\nbackend = TorchTensorRTAotBackend(config)\n```\n\n### TorchAO Backend\n\nTorchAO backend leverages PyTorch's AO (Accelerated Optimization) framework for model tuning.\n\n```python\nfrom aitune.torch.backend import TorchAOBackend\n\nbackend = TorchAOBackend()\n```\n\n### Torch Inductor Backend (JIT)\n\nTorch Inductor JIT backend uses PyTorch's Inductor compiler through `torch.compile` for model tuning.\n\n```python\nfrom aitune.torch.backend import TorchInductorJitBackend\n\nbackend = TorchInductorJitBackend()\n```\n\n### Torch Inductor Backend (AOT)\n\nTorch Inductor AOT backend uses PyTorch's AOT Inductor compiler to produce a compiled artifact that can be saved and loaded with AITune checkpoints.\n\n```python\nfrom aitune.torch.backend import TorchInductorAotBackend\n\nbackend = TorchInductorAotBackend()\n```\n\n### ONNXRuntime Backend\n\nONNXRuntime backend exports the selected PyTorch module to ONNX and runs inference through ONNX Runtime with CUDA or TensorRT execution providers.\n\n```python\nfrom aitune.torch.backend import ONNXRuntimeBackend, ONNXRuntimeBackendConfig, ONNXExecutionProvider\n\nconfig = ONNXRuntimeBackendConfig(execution_provider=ONNXExecutionProvider.CUDA)\nbackend = ONNXRuntimeBackend(config)\n```\n\n## Tune Strategies\n\nNVIDIA AITune provides different strategies for selecting the optimal backend configuration. The strategies align with a common interface for the tuning process.\n\nNot every backend can tune every model — each relies on different compilation technology with its own limitations (e.g., ONNX export for TensorRT, graph breaks in Torch Inductor, unsupported layers in TorchAO). Strategies control how AITune handles this.\n\n### FirstWinsStrategy\n\nTries backends in priority order and returns the first one that builds, validates correctness, and beats the Torch eager baseline by the configured threshold. If a backend fails or is slower than baseline, the strategy moves on to the next candidate instead of aborting.\n\n```python\nfrom aitune.torch.tune_strategy import FirstWinsStrategy\n\nstrategy = FirstWinsStrategy(backends=[TensorRTBackend(), TorchInductorJitBackend()])\n```\n\n### OneBackendStrategy\n\nUses exactly one backend, failing immediately with the original error if it cannot build. Use this when you have already validated that a backend works and want deterministic behavior. Unlike `FirstWinsStrategy` with a single backend, `OneBackendStrategy` surfaces the original exception rather than catching it.\n\n```python\nfrom aitune.torch.tune_strategy import OneBackendStrategy\n\nstrategy = OneBackendStrategy(backend=TensorRTBackend())\n```\n\n### MaxThroughputStrategy\n\nProfiles all compatible backends and selects the fastest one that beats the Torch eager baseline, falling back to eager when no user backend is faster. Use this when maximum throughput matters and you can afford longer tuning time.\n\n```python\nfrom aitune.torch.tune_strategy import MaxThroughputStrategy\n\nstrategy = MaxThroughputStrategy(backends=[TensorRTBackend(), TorchInductorJitBackend(), TorchEagerBackend()])\n```\n\n## Profiling with NVTX\n\nNVIDIA AITune includes NVTX (NVIDIA Tools Extension) annotations for profiling and debugging. NVTX marks key operations in the code, making them visible in profiling tools like NVIDIA Nsight Systems.\n\n**Note**: NVTX annotations are disabled by default to avoid overhead in production environments.\n\n### Enabling NVTX\n\nTo enable NVTX profiling, set the environment variable before running your script:\n\n```bash\nexport AITUNE_NVTX_EVENTS=1\npython your_script.py\n```\n\n### Using with Nsight Systems\n\nOnce enabled, you can profile your application with Nsight Systems:\n\n```bash\nAITUNE_NVTX_EVENTS=1 nsys profile -o output.nsys-rep -trace=cuda,nvtx,osrt python your_script.py\n```\n\nThe NVTX annotations will appear as colored regions in the timeline, helping you identify:\n\n* Backend inference calls (TensorRT, Torch-TensorRT, TorchAO, etc.)\n* Tuning operation\n* Performance bottlenecks\n\n\n\n## Hardware Metrics\n\nNVIDIA AITune can collect hardware metrics during tuning and inference, giving you visibility into resource utilization per module and backend. Metrics are collected in a background process and reported at program exit.\n\n**Note**: Hardware metrics collection is disabled by default to avoid overhead in production environments.\n\n### Enabling Hardware Metrics\n\nSet the environment variable before running your script:\n\n```bash\nexport AITUNE_HARDWARE_METRICS=1\npython your_script.py\n```\n\n### Collected Metrics\n\nThe following metrics are sampled continuously (every 100 ms by default) and aggregated per module and backend:\n\n| Category | Metrics |\n|---|---|\n| **GPU memory** (per device) | `cuda:N` used memory [GB] |\n| **GPU utilization** (per device) | `cuda:N` utilization mean / max [%] |\n| **GPU power** (per device) | `cuda:N` power mean / max [W] |\n| **Host CPU** | CPU utilization [%] |\n| **Host memory** | Used / free system memory |\n| **PyTorch allocator** | Allocated and reserved CUDA memory |\n\nGPU metrics require NVML (available when running on a system with NVIDIA drivers). If NVML is unavailable, only host and PyTorch metrics are collected.\n\n### Output\n\nAt program exit, AITune logs a summary table and writes a CSV file to the working directory.\n\nBy default a timestamped filename is used:\n\n```text\nhardware_metrics_20260402_153012.csv\n```\n\nTo write to a fixed path instead, set `AITUNE_HARDWARE_METRICS_PATH`:\n\n```bash\nexport AITUNE_HARDWARE_METRICS_PATH=hardware_metrics.csv\n```\n\nThe log summary looks like:\n\n```text\nINFO Hardware metrics summary:\n╒════════════════════════╤══════════════════════════════╤════════════╤════════════╤══════════════╤═════════════╤═════════════╤═════════════╕\n│ Module                 │ Backend                      │    Host    │   Cuda:0   │    Cuda:0    │   Cuda:0    │  Power [W]  │  Power [W]  │\n│                        │                              │  Mem [GB]  │  Mem [GB]  │  Util% mean  │  Util% max  │    mean     │     max     │\n╞════════════════════════╪══════════════════════════════╪════════════╪════════════╪══════════════╪═════════════╪═════════════╪═════════════╡\n│ CLIPTextModel          │ TensorRTBackend(             │   15.53    │    1.73    │     1.03     │      7      │    72.33    │   112.26    │\n│                        │     quantization_config=None │            │            │              │             │             │             │\n│                        │ )                            │            │            │              │             │             │             │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ Decoder                │ TensorRTBackend(             │   15.43    │    1.81    │      12      │     56      │   100.88    │   148.19    │\n│                        │     quantization_config=None │            │            │              │             │             │             │\n│                        │ )                            │            │            │              │             │             │             │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ Decoder                │ TensorRTBackend(             │   15.46    │    1.8     │    33.38     │     60      │   117.22    │   167.79    │\n│                        │     use_dynamo=False,        │            │            │              │             │             │             │\n│                        │     quantization_config=None │            │            │              │             │             │             │\n│                        │ )                            │            │            │              │             │             │             │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ Decoder                │ TorchInductorJitBackend()    │   15.53    │    1.7     │     3.12     │     85      │    85.92    │   179.21    │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ FluxTransformer2DModel │ TensorRTBackend(             │   14.36    │    1.79    │      0       │      0      │    67.84    │    71.79    │\n│                        │     quantization_config=None │            │            │              │             │             │             │\n│                        │ )                            │            │            │              │             │             │             │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ FluxTransformer2DModel │ TensorRTBackend(             │   14.35    │    1.79    │      0       │      0      │    63.46    │    63.46    │\n│                        │     use_dynamo=False,        │            │            │              │             │             │             │\n│                        │     quantization_config=None │            │            │              │             │             │             │\n│                        │ )                            │            │            │              │             │             │             │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ FluxTransformer2DModel │ TorchInductorJitBackend()    │   15.53    │    1.79    │     2.44     │     85      │    84.09    │   179.21    │\n├────────────────────────┼──────────────────────────────┼────────────┼────────────┼──────────────┼─────────────┼─────────────┼─────────────┤\n│ T5EncoderModel         │ TensorRTBackend(             │   16.65    │    1.77    │     1.76     │     85      │    70.57    │   179.21    │\n│                        │     quantization_config=None │            │            │              │             │             │             │\n│                        │ )                            │            │            │              │             │             │             │\n╘════════════════════════╧══════════════════════════════╧════════════╧════════════╧══════════════╧═════════════╧═════════════╧═════════════╛\n```\n\n### Combining with NVTX\n\nHardware metrics and NVTX profiling can be enabled together:\n\n```bash\nAITUNE_HARDWARE_METRICS=1 AITUNE_NVTX_EVENTS=1 nsys profile -o output.nsys-rep -trace=cuda,nvtx,osrt python your_script.py\n```\n\n## Examples\n\nWe offer comprehensive examples that showcase the utilization of NVIDIA AITune's diverse features. These examples are designed to elucidate the processes of tuning, profiling, testing, and deployment of models.\n\nFor detailed examples and step-by-step guides, please visit our [Examples Catalog](examples/README.md). The catalog includes practical implementations for various AI workloads including computer vision, natural language processing, speech recognition, and generative AI models.\n\n## Useful Links\n\n* [Changelog](CHANGELOG.md)\n* [Contributing](CONTRIBUTING.md)\n* [License](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-dynamo%2Faitune","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai-dynamo%2Faitune","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai-dynamo%2Faitune/lists"}