{"id":13717578,"url":"https://github.com/Media-Smart/volksdep","last_synced_at":"2025-05-07T07:31:46.964Z","repository":{"id":37693458,"uuid":"265798172","full_name":"Media-Smart/volksdep","owner":"Media-Smart","description":"volksdep is an open-source toolbox for deploying and accelerating PyTorch, ONNX and TensorFlow models with TensorRT.","archived":false,"fork":false,"pushed_at":"2021-02-05T02:47:21.000Z","size":78,"stargazers_count":285,"open_issues_count":6,"forks_count":43,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-04T00:13:45.597Z","etag":null,"topics":["acceleration","deploy","inference","jetson-nano","jetson-tx2","jetson-xavier","keras","onnx","python","pytorch","tensorflow","tensorrt"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Media-Smart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-21T08:43:10.000Z","updated_at":"2024-01-30T18:19:24.000Z","dependencies_parsed_at":"2022-08-23T23:41:13.347Z","dependency_job_id":null,"html_url":"https://github.com/Media-Smart/volksdep","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Media-Smart%2Fvolksdep","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Media-Smart%2Fvolksdep/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Media-Smart%2Fvolksdep/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Media-Smart%2Fvolksdep/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Media-Smart","download_url":"https://codeload.github.com/Media-Smart/volksdep/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224573505,"owners_count":17333804,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acceleration","deploy","inference","jetson-nano","jetson-tx2","jetson-xavier","keras","onnx","python","pytorch","tensorflow","tensorrt"],"created_at":"2024-08-03T00:01:24.315Z","updated_at":"2024-11-14T05:32:10.800Z","avatar_url":"https://github.com/Media-Smart.png","language":"Python","funding_links":[],"categories":["Pytorch \u0026 related libraries","Pytorch \u0026 related libraries｜Pytorch \u0026 相关库"],"sub_categories":["Other libraries:","Other libraries｜其他库:"],"readme":"## Introduction\nvolksdep is an open-source toolbox for deploying and accelerating PyTorch, ONNX and TensorFlow models with TensorRT.\n\n## Features\n- **Auto conversion and acceleration**\\\n    volksdep can accelerate PyTorch, ONNX and TensorFlow models using TensorRT with \n    only some few codes.\n\n- **Benchmark of throughput, latency and metric**\\\n    volksdep can generate benchmark of throughput, latency and metric with given model.\n\n## License\nThis project is released under [Apache 2.0 license](https://github.com/Media-Smart/volksdep/blob/master/LICENSE).\n\n## Installation\n### Requirements\n\n- Linux\n- Python 3.6 or higher\n- TensorRT 7.1.0.16 or higher\n- PyTorch 1.4.0 or higher\n- CUDA 10.2 or higher\n\nWe have tested the following versions of OS and softwares:\n\n- OS: Ubuntu 16.04.6 LTS\n- Python 3.6.9\n- TensorRT 7.1.3.4\n- PyTorch 1.4.0\n- CUDA: 10.2\n\n### Install volksdep\n\n1. If your platform is x86 or x64, you can create a conda virtual environment and activate it.\n\n```shell\nconda create -n volksdep python=3.6.9 -y\nconda activate volksdep\n```\n\n2. Install TensorRT following the [official instructions](https://developer.nvidia.com/tensorrt/)\n\n3. Install PyTorch and torchvision following the [official instructions](https://pytorch.org/)\n\n4. Setup.\n\n```shell\npip install \"git+https://github.com/Media-Smart/volksdep.git\"\n```\n\n## Known Issues\n1. PyTorch Upsample operation is supported with specified size, nearest mode and align_corners being None.\n\n## Usage\n### Convert\n\n#### PyTorch to TensorRT\n```shell\nimport torch\nimport torchvision\nfrom volksdep.converters import torch2trt\nfrom volksdep.calibrators import EntropyCalibrator2\nfrom volksdep.datasets import CustomDataset\n\ndummy_input = torch.ones(1, 3, 224, 224).cuda()\nmodel = torchvision.models.resnet18().cuda().eval()\n\n## build trt model with fp32 mode\ntrt_model = torch2trt(model, dummy_input)\n## build trt model with fp16 mode\n# trt_model = torch2trt(model, dummy_input, fp16_mode=True)\n## build trt model with int8 mode\n# trt_model = torch2trt(model, dummy_input, int8_mode=True)\n## build trt model with int8 mode and provided data using EntropyCalibrator2\n# dummy_calibrator = EntropyCalibrator2(CustomDataset(torch.randn(4, 3, 224, 224)))\n# trt_model = torch2trt(model, dummy_input, int8_mode=True, int8_calibrator=dummy_calibrator)\n```\nMore available arguments of torch2trt are detailed in \n[volksdep/converters/torch2trt.py](https://github.com/Media-Smart/volksdep/blob/master/volksdep/converters/torch2trt.py)\n\n#### ONNX to TensorRT\n```shell\nimport torch\nfrom volksdep.converters import onnx2trt\nfrom volksdep.calibrators import EntropyCalibrator2\nfrom volksdep.datasets import CustomDataset\n\nmodel = 'resnet18.onnx'\n\n## build trt model with fp32 mode\ntrt_model = onnx2trt(model)\n## build trt model with fp16 mode\n# trt_model = onnx2trt(model, fp16_mode=True)\n## build trt model with int8 mode\n# trt_model = onnx2trt(model, int8_mode=True)\n## build trt model with int8 mode and provided data using EntropyCalibrator2\n# dummy_calibrator = EntropyCalibrator2(CustomDataset(torch.randn(4, 3, 224, 224)))\n# trt_model = onnx2trt(model, int8_mode=True, int8_calibrator=dummy_calibrator)\n```\nMore available arguments of onnx2trt are detailed in \n[volksdep/converters/onnx2trt.py](https://github.com/Media-Smart/volksdep/blob/master/volksdep/converters/onnx2trt.py)\n\n#### Other frameworks to ONNX\n1. PyTorch to ONNX\n```shell\nimport torch\nimport torchvision\nfrom volksdep.converters import torch2onnx\n\ndummy_input = torch.ones(1, 3, 224, 224).cuda()\nmodel = torchvision.models.resnet18().cuda().eval()\ntorch2onnx(model, dummy_input, 'resnet18.onnx')\n```\nMore available arguments of torch2onnx are detailed in \n[volksdep/converters/torch2onnx.py](https://github.com/Media-Smart/volksdep/blob/master/volksdep/converters/torch2onnx.py)\n\n2. [TensorFlow to ONNX](https://github.com/onnx/tensorflow-onnx)\n\n3. [Keras to ONNX](https://github.com/onnx/keras-onnx)\n\n### Execute inference\n```shell\nwith torch.no_grad():\n    trt_output = trt_model(dummy_input)\n    print(trt_output.shape)\n```\n\n### Save and load\n#### Save\n```shell\nfrom volksdep.converters import save\n\nsave(trt_model, 'resnet18.engine')\n```\n#### Load\n```shell\nfrom volksdep.converters import load\n\ntrt_model = load('resnet18.engine')\n```\n\n### Benchmark\n#### PyTorch benchmark\n```shell\nimport torch\nimport torchvision\nfrom volksdep import benchmark\nfrom volksdep.calibrators import EntropyCalibrator, EntropyCalibrator2, MinMaxCalibrator\nfrom volksdep.datasets import CustomDataset\nfrom volksdep.metrics import Accuracy\n\nmodel = torchvision.models.resnet18()\n\n## simple benchmark, only test throughput and latency\nbenchmark(model, (1, 3, 224, 224), dtypes=['fp32', 'fp16', 'int8'])\n## benchmark with provided test dataset and metric\n# dummy_inputs = torch.randn(100, 3, 224, 224)\n# dummy_targets = torch.randint(0, 1001, size=(100,))\n# dummy_dataset = CustomDataset(dummy_inputs, dummy_targets)\n# metric = Accuracy()\n# benchmark(model, (1, 3, 224, 224), dataset=dummy_dataset, metric=metric)\n## benchmark with provided test dataset, metric and  data for int8 calibration\n# dummy_data = torch.randn(10, 3, 224, 224)\n# dummy_calibrators = [\n#     EntropyCalibrator(CustomDataset(dummy_data)),\n#     EntropyCalibrator2(CustomDataset(dummy_data)),\n#     MinMaxCalibrator(CustomDataset(dummy_data))\n# ]\n# dummy_dataset = CustomDataset(torch.randn(100, 3, 224, 224), torch.randint(0, 1001, size=(100,)))\n# metric = Accuracy()\n# benchmark(model, (1, 3, 224, 224), int8_calibrator=dummy_calibrators, dataset=dummy_dataset, metric=metric)\n```\n\n#### ONNX benchmark\n```shell\nimport torch\nimport torchvision\nfrom volksdep import benchmark\nfrom volksdep.calibrators import EntropyCalibrator, EntropyCalibrator2, MinMaxCalibrator\nfrom volksdep.datasets import CustomDataset\nfrom volksdep.metrics import Accuracy\n\nmodel = 'resnet18.onnx'\n\n## simple benchmark, only test throughput and latency\nbenchmark(model, (1, 3, 224, 224), framework='onnx', dtypes=['fp32', 'fp16', 'int8'])\n## benchmark with provided test dataset and metric\n# dummy_inputs = torch.randn(100, 3, 224, 224)\n# dummy_targets = torch.randint(0, 1001, size=(100,))\n# dummy_dataset = CustomDataset(dummy_inputs, dummy_targets)\n# metric = Accuracy()\n# benchmark(model, (1, 3, 224, 224), framework='onnx', dataset=dummy_dataset, metric=metric)\n## benchmark with provided test dataset, metric and  data for int8 calibration\n# dummy_data = torch.randn(10, 3, 224, 224)\n# dummy_calibrators = [\n#     EntropyCalibrator(CustomDataset(dummy_data)),\n#     EntropyCalibrator2(CustomDataset(dummy_data)),\n#     MinMaxCalibrator(CustomDataset(dummy_data))\n# ]\n# dummy_dataset = CustomDataset(torch.randn(100, 3, 224, 224), torch.randint(0, 1001, size=(100,)))\n# metric = Accuracy()\n# benchmark(model, (1, 3, 224, 224), framework='onnx', int8_calibrator=dummy_calibrators, dataset=dummy_dataset, metric=metric)\n```\n\nWe can define our own dataset and metric for int8 calibration and metric calculation.\n```shell\nimport numpy as np\nimport torch\nimport torchvision\nfrom volksdep.datasets import Dataset\nfrom volksdep.calibrators import EntropyCalibrator2\nfrom volksdep.metrics import Metric\nfrom volksdep import benchmark\n\n\nclass DatasetForCalibration(Dataset):\n    def __init__(self):\n        super(DatasetForCalibration, self).__init__()\n\n        self.dummy_inputs = torch.randn(10, 3, 224, 224)\n\n    def __getitem__(self, idx):\n        return self.dummy_inputs[idx]\n\n    def __len__(self):\n        return len(self.dummy_inputs)\n\n\nclass DatasetForMetric(Dataset):\n    def __init__(self):\n        super(DatasetForMetric, self).__init__()\n\n        self.dummy_inputs = torch.randn(100, 3, 224, 224)\n        self.dummy_targets = torch.randint(0, 1001, size=(100,))\n\n    def __getitem__(self, idx):\n        return self.dummy_inputs[idx], self.dummy_targets[idx]\n\n    def __len__(self):\n        return len(self.dummy_inputs)\n\n\nclass MyMetric(Metric):\n    def __init__(self):\n        super(MyMetric, self).__init__()\n\n    def __call__(self, preds, targets):\n        pred = np.argmax(preds, axis=-1)\n        acc = 1.0 * np.sum(pred == targets) / len(targets.flatten())\n\n        return acc\n\n    def __str__(self):\n        return 'my_metric'\n\n\ndummy_input = torch.randn(1, 3, 224, 224).cuda()\nmodel = torchvision.models.resnet18().cuda().eval()\ncalibrator = EntropyCalibrator2(DatasetForCalibration())\ndataset = DatasetForMetric()\nmetric = MyMetric()\n\nbenchmark(model, (1, 3, 224, 224), int8_calibrator=calibrator, dataset=dataset, metric=metric)\n```\n\n## Contact\nThis repository is currently maintained by Hongxiang Cai ([@hxcai](http://github.com/hxcai)), Yichao Xiong ([@mileistone](https://github.com/mileistone)).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMedia-Smart%2Fvolksdep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMedia-Smart%2Fvolksdep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMedia-Smart%2Fvolksdep/lists"}