https://github.com/redhawk989/onnx-multithreader

Run ONNX models in a multithreadded enviroment for lower CPU usage
https://github.com/redhawk989/onnx-multithreader

Last synced: about 1 year ago
JSON representation

Run ONNX models in a multithreadded enviroment for lower CPU usage

• Host: GitHub
• URL: https://github.com/redhawk989/onnx-multithreader
• Owner: RedHawk989
• License: gpl-3.0
• Created: 2023-06-22T22:31:31.000Z (about 3 years ago)
• Default Branch: main
• Last Pushed: 2023-10-09T14:04:31.000Z (over 2 years ago)
• Last Synced: 2025-02-12T17:46:58.252Z (over 1 year ago)
• Language: Python
• Size: 26.4 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# onnx-multithreader

Run ONNX models in a multithreaded environment for lower CPU usage

Increasing `inter_op_num_threads` for more FPS did not satisfy my usage requirements so I wrote this program to run 2 models and interweave the output for more efficient utilization.

In my testing, this method provided a 40x improvement in CPU usage for higher FPS needs, and can even be ~4 times more efficient than a normal ONNX inference script.

Found this repo useful? Give it a star!