{"id":17311394,"url":"https://github.com/dnth/supercharge-your-pytorch-image-models-blogpost","last_synced_at":"2025-08-03T06:30:57.162Z","repository":{"id":256550611,"uuid":"855725242","full_name":"dnth/supercharge-your-pytorch-image-models-blogpost","owner":"dnth","description":"Supercharge Your PyTorch Image Models: Bag of Tricks to 8x Faster Inference with ONNX Runtime \u0026 Optimizations","archived":false,"fork":false,"pushed_at":"2024-10-04T16:51:37.000Z","size":65042,"stargazers_count":20,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-02T07:45:55.901Z","etag":null,"topics":["computer-vision","onnx","onnxruntime","onnxruntime-gpu","pytorch","tensorrt","timm"],"latest_commit_sha":null,"homepage":"https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dnth.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-11T11:12:17.000Z","updated_at":"2024-11-20T10:07:41.000Z","dependencies_parsed_at":"2024-09-11T16:43:45.963Z","dependency_job_id":"a466a248-39f1-44f1-ae88-16c05fef5310","html_url":"https://github.com/dnth/supercharge-your-pytorch-image-models-blogpost","commit_stats":null,"previous_names":["dnth/timm_onnx_tensort","dnth/supercharge-your-pytorch-image-models-blogpost"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnth%2Fsupercharge-your-pytorch-image-models-blogpost","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnth%2Fsupercharge-your-pytorch-image-models-blogpost/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnth%2Fsupercharge-your-pytorch-image-models-blogpost/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dnth%2Fsupercharge-your-pytorch-image-models-blogpost/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dnth","download_url":"https://codeload.github.com/dnth/supercharge-your-pytorch-image-models-blogpost/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":227931112,"owners_count":17842979,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","onnx","onnxruntime","onnxruntime-gpu","pytorch","tensorrt","timm"],"created_at":"2024-10-15T12:40:26.498Z","updated_at":"2024-12-03T13:49:04.800Z","avatar_url":"https://github.com/dnth.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![PyTorch to ONNX-TensorRT](https://dicksonneoh.com/images/portfolio/supercharge_your_pytorch_image_models/post_image.png)\n\nThis repository contains code to optimize PyTorch image models using ONNX Runtime and TensorRT, achieving up to 8x faster inference speeds. Read the full blog post [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models/).\n\n\n## Installation\nCreate and activate a conda environment:\n\n```bash\nconda create -n supercharge_timm_tensorrt python=3.11\nconda activate supercharge_timm_tensorrt\n```\n Install required packages:\n\n\n```bash\npip install timm\npip install onnx\npip install onnxruntime-gpu==1.19.2\npip install cupy-cuda12x\npip install tensorrt==10.1.0 tensorrt-cu12==10.1.0 tensorrt-cu12-bindings==10.1.0 tensorrt-cu12-libs==10.1.0\n```\n\nInstall CUDA dependencies:\n```bash\nconda install -c nvidia cuda=12.2.2 cuda-tools=12.2.2 cuda-toolkit=12.2.2 cuda-version=12.2 cuda-command-line-tools=12.2.2 cuda-compiler=12.2.2 cuda-runtime=12.2.2\n```\n\nInstall cuDNN:\n```bash\nconda install cudnn==9.2.1.18\n```\n\nSet up library paths:\n```bash\nexport LD_LIBRARY_PATH=\"/home/dnth/mambaforge-pypy3/envs/supercharge_timm_tensorrt/lib:$LD_LIBRARY_PATH\"\nexport LD_LIBRARY_PATH=\"/home/dnth/mambaforge-pypy3/envs/supercharge_timm_tensorrt/lib/python3.11/site-packages/tensorrt_libs:$LD_LIBRARY_PATH\"\n```\n\n## Running the code\n\nThe following codes correspond to the steps in the blog post.\n\n### Load timm model and run inference:\n   ```bash\n   python 00_load_and_infer.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models/#-load-and-infer)\n\n### PyTorch latency benchmark:\n   ```bash\n   python 01_pytorch_latency_benchmark.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-baseline-latency)\n\n### Convert model to ONNX:\n   ```bash\n   python 02_convert_to_onnx.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-convert-to-onnx)\n\n### ONNX Runtime CPU inference:\n   ```bash\n   python 03_onnx_cpu_inference.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-onnx-runtime-on-cpu)\n\n### ONNX Runtime CUDA inference:\n   ```bash\n   python 04_onnx_cuda_inference.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-onnx-runtime-on-cuda)\n\n### ONNX Runtime TensorRT inference:\n   ```bash\n   python 05_onnx_trt_inference.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-onnx-runtime-on-tensorrt)\n\n### Export preprocessing to ONNX:\n   ```bash\n   python 06_export_preprocessing_onnx.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-bake-pre-processing-into-onnx)\n\n### Merge preprocessing and model ONNX:\n   ```bash\n   python 07_onnx_compose_merge.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-bake-pre-processing-into-onnx)\n\n### Run inference on merged model:\n   ```bash\n   python 08_inference_merged_model.py\n   ```\nRead more [here](https://dicksonneoh.com/portfolio/supercharge_your_pytorch_image_models//#-bake-pre-processing-into-onnx)\n\n### Run inference on video:\n   ```bash\n   python 09_video_inference.py sample.mp4 output.mp4 --live \n   ```\n\n\n\nhttps://github.com/user-attachments/assets/1a25dd6e-3512-475a-9541-29e836022bb5\n\n\n\n\n\n\u003c!-- # Pytorch to ONNX-TensorRT\n\nThis repository contains a script to convert a PyTorch model to ONNX format and then to TensorRT format.\n\n## Prerequisites\n\n- PyTorch\n- ONNX\n- TensorRT\n\n## Installation\nFor simplicity, I'll use a conda environment with Python 3.11.\n\nSetup conda environment:\n```bash\nconda create -n pt-to-onnx-tensorrt python=3.11\nconda activate pt-to-onnx-tensorrt\n```\n\n\n1. Install CUDA components:\n   ```bash\n   conda install -y -c nvidia cuda=12.2.2 cuda-tools=12.2.2 cuda-toolkit=12.2.2 cuda-version=12.2 cuda-command-line-tools=12.2.2 cuda-compiler=12.2.2 cuda-runtime=12.2.2\n   ```\n\n2. Install cuDNN:\n   ```bash\n   conda install cudnn==9.2.1.18\n   ```\n\n3. Install ONNX Runtime GPU:\n   ```bash\n   pip install -U onnxruntime-gpu==1.19.2\n   ```\n4. Install TensorRT:\n   ```bash\n   pip install tensorrt==10.1.0 tensorrt-cu12==10.1.0 tensorrt-cu12-bindings==10.1.0 tensorrt-cu12-libs==10.1.0\n   ```\n\n5. Install TIMM:\n   ```bash\n   pip install timm, onnx, cupy\n   ```\n\n6. Set up library paths:\n   ```bash\n   export LD_LIBRARY_PATH=\"/path/to/your/conda/env/lib:$LD_LIBRARY_PATH\"\n   export LD_LIBRARY_PATH=\"/path/to/your/conda/env/lib/python3.11/site-packages/tensorrt_libs:$LD_LIBRARY_PATH\"\n   ```\n   Note: Adjust the paths according to your Conda environment location.\n\n\n## Notebooks\nBenchmark notebooks:\n- [Benchmark TIMM](./notebooks/benchmark_timm.ipynb)\n- [Benchmark ONNX Runtime CPU](./notebooks/benchmark_onnxruntime_cpu.ipynb)\n- [Benchmark ONNX Runtime GPU](./notebooks/benchmark_onnxruntime_gpu.ipynb)\n- [Benchmark TensorRT](./notebooks/benchmark_tensorrt.ipynb)\n\n\nConversion notebooks:\n- [Pytorch to ONNX](./notebooks/pytorch_to_onnx.ipynb)\n- [ONNX to TensorRT](./notebooks/onnx_to_tensorrt.ipynb)     --\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdnth%2Fsupercharge-your-pytorch-image-models-blogpost","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdnth%2Fsupercharge-your-pytorch-image-models-blogpost","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdnth%2Fsupercharge-your-pytorch-image-models-blogpost/lists"}