{"id":15172771,"url":"https://github.com/parlaynu/inference-onnx","last_synced_at":"2025-04-13T09:31:01.192Z","repository":{"id":209589829,"uuid":"724462818","full_name":"parlaynu/inference-onnx","owner":"parlaynu","description":"Export PyTorch to ONNX and run inference in containerized environments.","archived":false,"fork":false,"pushed_at":"2023-12-02T07:49:32.000Z","size":33,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T01:12:11.402Z","etag":null,"topics":["deepstream","docker","gstreamer","jetson-nano","nvargus","nvidia-gpu","onnx","onnxruntime","picamera2","python","pytorch","pyzmq","raspberry-pi-4","zeromq"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/parlaynu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-28T05:57:13.000Z","updated_at":"2024-10-16T18:42:58.000Z","dependencies_parsed_at":"2023-11-28T07:25:00.674Z","dependency_job_id":"9af17a9a-15eb-420c-8c1a-112bf3e9c1e6","html_url":"https://github.com/parlaynu/inference-onnx","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"f116d77bc706e1ff927d1c7b53bb4efb4a6a6c11"},"previous_names":["parlaynu/inference-onnx"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Finference-onnx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Finference-onnx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Finference-onnx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/parlaynu%2Finference-onnx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/parlaynu","download_url":"https://codeload.github.com/parlaynu/inference-onnx/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248690759,"owners_count":21146200,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepstream","docker","gstreamer","jetson-nano","nvargus","nvidia-gpu","onnx","onnxruntime","picamera2","python","pytorch","pyzmq","raspberry-pi-4","zeromq"],"created_at":"2024-09-27T10:04:18.624Z","updated_at":"2025-04-13T09:31:00.857Z","avatar_url":"https://github.com/parlaynu.png","language":"Python","readme":"# Inference Using ONNX\n\nThis repository has tools and guidelines for exporting PyTorch models to ONNX and running classification inference using the exported model. \n\nThe tools include:\n\n* exporting from pytorch to onnx\n* running inference using the exported onnx model\n* camera image servers for RaspberryPi and Jetson Nano\n\nIt also has tools for building docker images to run the applications in various platforms. The available combinations of platform\nand tool are shown in the table below.\n\n| Tool           |  x86_64  | Jetson Nano  | RaspberryPi4 |\n| -------------- | -------- | -------------| ------------ |\n| torch2onnx     |   X      |              |              |\n| infer-onnx     |   X      |   X          |   X          |\n| nvargus-server |          |   X          |              |\n| picam2-server  |          |              |   X          |\n\nThe `tools` directory contains the source code in python for the tools and the `platforms` directory has the tools\nto build the docker images and environments for each of the supported platforms.\n\nI wasn't able to run the camera software within a container as there are a lot of system dependencies. It might be\npossible to make it work, but I gave up and instead created them as servers which can be connected to from within\nan containerised environment to access images. This also means that these camera servers can serve images to processes\nthat are on other machines. ZeroMQ, specifically pyzmq, is used for the networking.\n\n## The Tools\n\n### Exporting PyTorch to ONNX\n\nThis is currently a very simple script. It downloads pretrained PyTorch models and exports to ONNX. It currently\ndoesn't even support loading custom model weights, but that will come soon... probably.\n\nTo run it:\n\n    $ ./torch2onnx.py -h\n    usage: torch2onnx.py [-h] [--bsize BSIZE] [--resolution RESOLUTION] [--channels CHANNELS] model [outdir]\n    \n    positional arguments:\n      model                 model architecture\n      outdir                output directory to write models to\n      \n    options:\n      -h, --help            show this help message and exit\n      --bsize BSIZE         batch size\n      --resolution RESOLUTION\n                            the image resolution for the model: WIDTHxHEIGHT\n      --channels CHANNELS   number of input channels\n\nA minimal run of the tools is:\n\n    $ ./torch2onnx.py --resolution 224x224 resnet18\n    Downloading: \"https://download.pytorch.org/models/resnet18-f37072fd.pth\" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth\n    100.0%\n      batch size: 1\n      image size: 224x224\n    num channels: 3\n     num classes: 1000\n    exporting to ./local/models/resnet18.onnx\n\nThis defaults to a batch size of 1, extracts the number of channels and classes from the model itself, but requires\nthe input image resolution to be specified on the command line.\n\nThere are no dynamic axes in the exported model, only static. Although, it would be a simple thing to change if you\nwanted to.\n\n### Running Inference\n\nThe tool `classify-onnx.py` runs inference on the exported ONNX model. The full usage is:\n\n    $ ./classify-onnx.py -h\n    usage: classify-onnx.py [-h] [-l LIMIT] [-r RATE] [-c] model dataspec\n    \n    positional arguments:\n      model                 path to the onnx model file to load\n      dataspec              the data source specification\n      \n    options:\n      -h, --help            show this help message and exit\n      -l LIMIT, --limit LIMIT\n                            maximum number of images to process\n      -r RATE, --rate RATE  requests per second\n      -c, --force-cpu       force use the cpu even if there is a gpu\n\nThe `model` parameter is the path to the ONNX model to load and use for inference. Check the platform\nspecific notes for instructions on how to access a model from within the container.\n\nThe dataspec can be a filesystem path to a directory of images or a URL to a camera server. The camera\nserver URL that the server is listening on printed out when the server starts and can be used directly \nhere. See the camera server sections for details.\n\nNote that if you use the scripts provided to run the inference container, you can access these servers\nif they are bound to 127.0.0.1 as the containers are started with host networking.\n\n### RaspberryPi Camera Server\n\nThe RaspberryPi camera server runs like this:\n\n    $ ./picam2-server.py -h\n    usage: picam2-server.py [-h] [-i] [-a] [-p PORT] [-r FPS] [-m MODE] [-v] [--hflip] [--vflip]\n    \n    options:\n      -h, --help            show this help message and exit\n      -i, --ipc             use the IPC transport\n      -a, --all             listen on all interfaces\n      -p PORT, --port PORT  port to listen on\n      -r FPS, --fps FPS     camera frame rate\n      -m MODE, --mode MODE  the camera mode\n      -v, --preview         enable previewing\n      --hflip               flip the image horizontally\n      --vflip               flip the image vertically\n\nTo find the available modes for your camera, run the `picam2-info.py` tool. It is possible to run\nusing 'ipc' as the protocol, however, this won't be accessible from other hosts or within the container as they are\ncurrently launched.\n\nThe simplest run is:\n\n    $ ./picam2-server.py \n    listening on tcp://127.0.0.1:8089\n\nTo listen on all addresses on the server:\n\n    $ ./picam2-server.py -a\n    listening on tcp://192.168.1.31:8089\n\nYou can use the listening URLs directly in the inference application to specify a data source.\n\nA lot of debug information is printed out, some of which is from the libcamera subsystem and can't be\ncontrolled. It prints out the running configuration of the camera and then sits and waits for a request for\nan image.\n\n    {'buffer_count': 2,\n     'colour_space': \u003clibcamera.ColorSpace 'sYCC'\u003e,\n     'controls': {'FrameDurationLimits': (33333, 500000),\n                  'NoiseReductionMode': \u003cNoiseReductionModeEnum.HighQuality: 2\u003e},\n     'display': None,\n     'encode': None,\n     'lores': None,\n     'main': {'format': 'BGR888',\n              'framesize': 6220800,\n              'size': (1920, 1080),\n              'stride': 5760},\n     'queue': False,\n     'raw': {'format': 'SBGGR10_CSI2P',\n             'framesize': 2592000,\n             'size': (1920, 1080),\n             'stride': 2400},\n     'sensor': {'bit_depth': 10, 'output_size': (1920, 1080)},\n     'transform': \u003clibcamera.Transform 'identity'\u003e,\n     'use_case': 'still'}\n\n\n### Jetson Nano Camera Server\n\nThe NvArgus camera server runs like this:\n\n    $ ./nvargus-server.py -h\n    usage: nvargus-server.py [-h] [-i] [-a] [-p PORT] [-r FPS] [--hflip] [--vflip]\n                             [-c]\n                             \n    optional arguments:\n      -h, --help            show this help message and exit\n      -i, --ipc             use the IPC transport\n      -a, --all             listen on all interfaces\n      -p PORT, --port PORT  port to listen on\n      -r FPS, --fps FPS     camera frame rate\n      --hflip               flip the image horizontally\n      --vflip               flip the image vertically\n      -c, --centre          crop the centre square of the image\n\nThe simplest run is:\n\n    $ ./nvargus-server.py \n    listening on tcp://127.0.0.1:8089\n\nTo listen on all addresses on the server:\n\n    ./nvargus-server.py -a\n    listening on tcp://192.168.1.39:8089\n\nYou can use the listening URLs directly in the inference application to specify a data source.\n\nA lot of debug information is printed out, some of which is from the argus subsystem and can't be\ncontrolled. One useful thing it does print is the available modes for your camera. If you run it once\nwith just the defaults, you will get this information and then be able to choose the mode you want\nto use in a subsequent run.\n\nThe modes for my camera are like this (first one is mode 0):\n\n    GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;\n    GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;\n    GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;\n    GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;\n    GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;\n    GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;\n\nIt prints out the running configuration of the camera and then sits and waits for a request for\nan image.\n\n    GST_ARGUS: Running with following settings:\n       Camera index = 0 \n       Camera mode  = 2 \n       Output Stream W = 1920 H = 1080 \n       seconds to Run    = 0 \n       Frame Rate = 29.999999 \n    GST_ARGUS: Setup Complete, Starting captures for 0 seconds\n\nThere is also a `nvargus-preview.py` that will run the camera and display the output on the jetson's\nmonitor output.\n\n## The Platforms\n\nUnder the `platform` directory, there is a directory for each platform supported, and inside each platform,\nthere is a directory for each tool for that platform.\n\nThe README.md has platform specific setup notes and also includes instructions for building and launching the\ncontainer. Each has a `build.sh` script that builds the docker image, and a `run-latest.sh` which launches \nthe latest image as a container and starts an interactive bash session from which you can run the tools.\n\nThe script launches the container with all the correct options needed to run the tools inside it.\n\nThe scripts are all very simple and along with the Dockerfiles, should be very easy to follow and see what's\nhappening.\n\nThe exception to the above are the camera servers on the RaspberryPi and Jetson Nano as these can't run in \na containerized environment. The platform specific README.md files have full instructions on how to setup the\nenvironment for these cameras.\n\n\n\n\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparlaynu%2Finference-onnx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fparlaynu%2Finference-onnx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fparlaynu%2Finference-onnx/lists"}