{"id":13643280,"url":"https://github.com/shouxieai/infer","last_synced_at":"2025-04-05T14:04:49.634Z","repository":{"id":112413952,"uuid":"608946608","full_name":"shouxieai/infer","owner":"shouxieai","description":"A new tensorrt integrate. Easy to integrate many tasks","archived":false,"fork":false,"pushed_at":"2023-04-02T07:15:15.000Z","size":48797,"stargazers_count":418,"open_issues_count":20,"forks_count":84,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-29T13:09:31.539Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shouxieai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-03T03:31:51.000Z","updated_at":"2025-03-26T10:37:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"0c839987-bfae-4a73-b240-af8f164c876a","html_url":"https://github.com/shouxieai/infer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shouxieai%2Finfer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shouxieai%2Finfer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shouxieai%2Finfer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shouxieai%2Finfer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shouxieai","download_url":"https://codeload.github.com/shouxieai/infer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345850,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T01:01:45.119Z","updated_at":"2025-04-05T14:04:49.606Z","avatar_url":"https://github.com/shouxieai.png","language":"Cuda","funding_links":[],"categories":["Lighter and Deployment Frameworks","Applications"],"sub_categories":[],"readme":"# New tensorrt package, easy to integrate many tasks\n- Easily implement producer-consumer models for various tasks and perform high-performance inference\n- No complex packaging, no coupling!\n\n# For the Yolo-Demo\n- Currently supports Yolo series 3/4/5/x/7/8\n- YoloV8-Segment is supported\n- 🚀 Pre-processing about 1ms\n- 🚀 Post-processing about 0.5ms\n![](bus.jpg)\n\n# Description\n- cpm.hpp Producer-consumer model\n    - For direct inference tasks, cpm.hpp can be turned into an automatic multi-batch producer-consumer model\n- infer.hpp A repackaging of tensorRT. Simple interface\n- yolo.hpp Wrapper for yolo tasks. Based on infer.hpp\n\n### Inference flow of trt\n### step1 Compile the model, e.g.\n`trtexec --onnx=yolov5s.onnx --saveEngine=yolov5s.engine`\n\n### step2: Use infer inference\n```c++\nmodel = trt::load(\"yolov5s.engine\");\n... preprocess ...\n\n// Configure the dynamic batch size.\nauto dims = model-\u003estatic_dims();\ndims[0] = batch;\nmodel-\u003eset_run_dims(dims);\nmodel-\u003eforward({input_device, output_device}, stream);\n\n... postprocess ...\n```\n\n### step2: Use yolo inference\n```c++\ncv::Mat image = cv::imread(\"image.jpg\");\nauto model = yolo::load(\"yolov5s.engine\");\nauto objs = model-\u003eforward(yolo::Image(image.data, image.cols, image.rows));\n// use objs to draw to image. \n```\n\n\n# Use of CPM (wrapping the inference as producer-consumer)\n```c++\ncpm::Instance\u003cyolo::BoxArray, yolo::Image, yolo::Infer\u003e cpmi;\ncpmi.start([]{\n    return yolo::load(\"yolov5s.engine\", yolo::Type::V5);\n}, batch);\n\nauto result_futures = cpmi.commits(images);\nfor(auto\u0026 fut : result_futures){\n    auto objs = fut.get();\n    ... process ...\n}\n```\n# Reference\n- [💡Video: 1. How to use TensorRT efficiently](https://www.bilibili.com/video/BV1F24y1h7LW)\n- [😁Video: 2. Feeling of using Infer](https://www.bilibili.com/video/BV1B24y137nW)\n- [💕Video: 3. Instance segmentation and detection of YoloV8](https://www.bilibili.com/video/BV1SY4y1C7E2)\n- [😍Video: 4. Static batch \u0026 Dynamic batch](https://www.bilibili.com/video/BV15Y41167B5)\n- [🌻TensorRT_Pro](https://github.com/shouxieai/tensorRT_Pro)\n- [🔭KIWI: Enable AI with One Click!](https://www.shouxieai.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshouxieai%2Finfer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshouxieai%2Finfer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshouxieai%2Finfer/lists"}