{"id":15062608,"url":"https://github.com/ivangolt/yolo_triton_tensorrt","last_synced_at":"2026-01-03T03:32:43.244Z","repository":{"id":254709313,"uuid":"846648570","full_name":"ivangolt/yolo_triton_tensorrt","owner":"ivangolt","description":"Repository of implementation yolo with triton inference client and tensorrt optimization. Model deployment with FastAPI","archived":false,"fork":false,"pushed_at":"2024-09-09T17:04:20.000Z","size":2320,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-22T04:33:13.329Z","etag":null,"topics":["fastapi","grafana","prometheus","streamlit","tensorrt","triton","yolov8"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivangolt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-23T16:49:15.000Z","updated_at":"2024-09-09T17:04:24.000Z","dependencies_parsed_at":"2024-09-06T00:23:20.347Z","dependency_job_id":"56c4e0a0-c543-4251-8c4c-44b32a9f008e","html_url":"https://github.com/ivangolt/yolo_triton_tensorrt","commit_stats":null,"previous_names":["ivangolt/yolo_triton_tensorrt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangolt%2Fyolo_triton_tensorrt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangolt%2Fyolo_triton_tensorrt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangolt%2Fyolo_triton_tensorrt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivangolt%2Fyolo_triton_tensorrt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivangolt","download_url":"https://codeload.github.com/ivangolt/yolo_triton_tensorrt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243741179,"owners_count":20340417,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","grafana","prometheus","streamlit","tensorrt","triton","yolov8"],"created_at":"2024-09-24T23:43:33.693Z","updated_at":"2026-01-03T03:32:38.199Z","avatar_url":"https://github.com/ivangolt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Yolo triton tensorrt Fast API Streamlit\n# Overview\nThis repository  provides an ensemble model that combines a YOLOv8 model exported from the [Ultralytics](https://github.com/ultralytics/ultralytics) repository with NMS (Non-Maximum Suppression) post-processing for deployment on the Triton Inference Server using a TensorRT backend, deployment rest api service in FastAPI and frontend in streamlit.\n\n\nFor more information about Triton's Ensemble Models, see their documentation on [Architecture.md](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/architecture.md) and some of their [preprocessing examples](https://github.com/triton-inference-server/python_backend/tree/main/examples/preprocessing).\n\n# Directory Structure\n```\n├── Dockerfile                   # Docker file to build Triton image\n├── LICENSE\n├── notebooks                    # notebooks for example\n├── utils\n|   ├── load_model.py            # load model and convert to onnx and tensorrt format and move them to /models repository\n|   ├── yolo_classes.py          # yolo classes names\n├── data\n|\n├── app\n|   ├── Dockerfile               # FastAPI Dockerfile\n|   ├── main.py                  # Main app with FastAPI initializing\n|\n├── frontend\n|   ├── app.py                   # web ui application\n|   ├── Dockerfile               # Dockerfile for streamlit service\n|\n├── triton                       # triton model path\n|   ├── client.py                # triton client in python\n├── models                       \n│   ├── postprocess\n│   │   ├── 1\n│   │   │   ├── model.py\n│   │   └── config.pbtxt\n│   ├── yolov8_ensemble\n│   │   ├── 1\n│   │   │   └── model.plan\n│   │   └── config.pbtxt\n│   └── yolov8_tensorrt\n│       ├── 1\n│       │   └── model.plan\n│       └── config.pbtxt\n├── docker-compose.yaml          # docker compose for running all parts of application\n└── README.md\n```\n\n# Triton client\n1. Install [Ultralytics](https://github.com/ultralytics/ultralytics) and TritonClient\n```\npip install ultralytics tritonclient[all] \n```\n2. (Optional): Update the Score and NMS threshold in [models/postprocess/1/model.py](models/postprocess/1/model.py#L59)\n3. (Optional): Update the [models/yolov8_ensemble/config.pbtxt](models/yolov8_ensemble/config.pbtxt) file if your input resolution has changed.\n4. Build the Docker Container for Triton Inference:\n```\nDOCKER_NAME=\"yolov8-triton\"\ndocker build -t $DOCKER_NAME .\n```\n5. Load onnx Yolo Model \n```\npython ./utils/load_model.py --model_name {model_name} (e.g. yolov8m.pt)\n```\n6. Inside the container of Triton Inference Server, use the `trtexec` tool to convert the YOLOv8 ONNX model to a TensorRT engine file. \n```\n/usr/src/tensorrt/bin/trtexec --onnx=/path/to/your/folder/model.onnx --saveEngine=/models/yolov8.engine --fp16 --shapes=images:1x3x640x640\n```\n   Rename the yolov8.engine file to `model.plan` and place it under the `/models/yolov8_tensorrt/1` directory  and the `/models/yolov8_ensemble/1` directory (see directory structure above).\n\n7. Run Triton Inference Server:\n```\nDOCKER_NAME=\"yolov8-triton\"\ndocker run --gpus all \\\n    -it --rm \\\n    --name triton\n    --net=host \\\n    -p 8000:8000                            # grpc\n    -p 8001:8001                            # http  \n    -p 8002:8002\n    -v {abs_path_to_your_models}:/models \\\n    $DOCKER_NAME\n```\n8. Run the script with `python ./clients/client.py`. The inferred overlay image will be written to `./results/output.jpg`.\n\n# FastAPI\n\n1. Overview\n\nThis API provides an endpoint for performing object detection on images using a YOLO-based model deployed on a Triton Inference Server. Users can upload an image, and the API will return the image with bounding boxes drawn around the detected objects.\n\n\n2. Endpoints\n\nGET /: A simple root endpoint that returns a greeting message.\n\nPOST /predict/: The primary endpoint that accepts an image file, processes it using a YOLO model, and returns the image with detected objects highlighted.\n\n\nFor testing endpoint of FastAPI service\n\n``` curl -X 'POST' \\\n  'http://127.0.0.1:8000/predict/' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: multipart/form-data' \\\n  -F 'file=@{image_path};type=image/jpeg'\n\n```\n\n# Streamlit \n\n1. Overview\n\nThis Streamlit application provides an interface for performing object detection on images using the YOLOv8 model. Users can upload an image, and the app will display the image with detected objects highlighted. \n\n\nRun the Streamlit app with the command\n\n```\nstreamlit run app.py\n\n```\n\n# Run aplication as service\n\nRun building docker-compose.yaml\n\n```\ndocker-compose up -d\n\n```\n\nafter that go to \"http://localhost:8501\"\n\nExample:\n\n![](docs/image.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivangolt%2Fyolo_triton_tensorrt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivangolt%2Fyolo_triton_tensorrt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivangolt%2Fyolo_triton_tensorrt/lists"}