{"id":30580535,"url":"https://github.com/roboflow/single_artifact_benchmarking","last_synced_at":"2025-09-13T19:46:46.430Z","repository":{"id":308884757,"uuid":"1023913073","full_name":"roboflow/single_artifact_benchmarking","owner":"roboflow","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-23T17:11:25.000Z","size":54,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-28T12:36:02.716Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/roboflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-21T22:40:28.000Z","updated_at":"2025-08-06T13:45:04.000Z","dependencies_parsed_at":"2025-08-08T14:05:49.475Z","dependency_job_id":null,"html_url":"https://github.com/roboflow/single_artifact_benchmarking","commit_stats":null,"previous_names":["roboflow/single_artifact_benchmarking"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/roboflow/single_artifact_benchmarking","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fsingle_artifact_benchmarking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fsingle_artifact_benchmarking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fsingle_artifact_benchmarking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fsingle_artifact_benchmarking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/roboflow","download_url":"https://codeload.github.com/roboflow/single_artifact_benchmarking/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/roboflow%2Fsingle_artifact_benchmarking/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275020167,"owners_count":25391650,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-13T02:00:10.085Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-29T05:36:44.942Z","updated_at":"2025-09-13T19:46:46.384Z","avatar_url":"https://github.com/roboflow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Standardized Object Detection Benchmarking\n\n## Problem\n\nCurrent object detection benchmarking practices suffer from significant inconsistencies that compromise the reliability of reported performance metrics. Typically, researchers report mAP values from their research code, then export models to ONNX format and compile with fp16 TensorRT to report latency measurements. This approach introduces several sources of error:\n\n1. **Precision Compatibility**: Some models do not function correctly when compiled to fp16 precision\n2. **Postprocessing Overhead**: Complex postprocessing operations significantly impact model performance but are inconsistently handled across implementations\n3. **Measurement Methodology**: Inconsistent reporting between raw `trtexec` outputs and Python session measurements\n4. **Thermal Throttling**: Inadequate control for GPU power throttling due to thermal saturation, leading to unreproducible latency measurements\n\n## Solution\n\nThis framework provides an optimized TensorRT Python implementation that translates directly from ONNX graphs to latency/mAP pairs without leveraging complex postprocessing for any model. The implementation addresses the identified issues through:\n\n- **Throttling Monitoring**: Active detection of GPU thermal throttling to determine measurement reliability\n- **Thermal Management**: Insertion of cooling buffers between subsequent inference calls to reduce throttling effects\n- **Hosted Model Repository**: Centralized hosting of ONNX graphs to ensure model availability and reproducibility\n- **Standardized Export**: Consistent model export methodology across architectures\n\n## Model Export Standards\n\nONNX graphs are obtained directly from the original author repositories for each model type. For YOLO models specifically, export is performed using the command:\n\n```\nyolo export format=onnx nms=True conf=0.001\n```\n\n## Technical Implementation\n\nA notable distinction from the D-FINE implementation is the inclusion of CUDA graph support. While CUDA graphs are straightforward to implement with `trtexec`, they present additional complexity in Python environments. However, they provide meaningful performance improvements for certain model architectures, justifying their inclusion in this framework.\n\n## Usage\n\nTo run the benchmark:\n\n1. Install dependencies: `pip install -r requirements.txt`\n2. Execute: `python3 benchmark_all.py \u003cpath to coco val dir\u003e \u003cpath to coco val annotations\u003e`\n\n## Contributions\n\nContributions of new models to the benchmark suite are welcome. Please submit model additions by opening a pull request to the repository.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fsingle_artifact_benchmarking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Froboflow%2Fsingle_artifact_benchmarking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Froboflow%2Fsingle_artifact_benchmarking/lists"}