{"id":48863229,"url":"https://github.com/gpustack/benchmark-runner","last_synced_at":"2026-04-15T17:05:39.963Z","repository":{"id":335405942,"uuid":"1144113401","full_name":"gpustack/benchmark-runner","owner":"gpustack","description":null,"archived":false,"fork":false,"pushed_at":"2026-03-05T09:13:33.000Z","size":445,"stargazers_count":1,"open_issues_count":1,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-05T12:51:45.562Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gpustack.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-28T10:00:37.000Z","updated_at":"2026-03-05T09:13:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/gpustack/benchmark-runner","commit_stats":null,"previous_names":["aiwantaozi/benchmark-runner"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/gpustack/benchmark-runner","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fbenchmark-runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fbenchmark-runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fbenchmark-runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fbenchmark-runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gpustack","download_url":"https://codeload.github.com/gpustack/benchmark-runner/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gpustack%2Fbenchmark-runner/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31851086,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-15T15:24:51.572Z","status":"ssl_error","status_checked_at":"2026-04-15T15:24:39.138Z","response_time":63,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-15T17:05:39.269Z","updated_at":"2026-04-15T17:05:39.954Z","avatar_url":"https://github.com/gpustack.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Benchmark Runner\n================\n\nBenchmark Runner is a thin wrapper around GuideLLM that provides a simplified CLI,\ncustom progress reporting, and ShareGPT dataset preparation for benchmarking\ngenerative models.\n\nWhat it adds\n------------\n- A streamlined `benchmark-runner` CLI focused on benchmark and config commands.\n- Optional server-side progress updates during benchmarks.\n- ShareGPT dataset conversion to GuideLLM-compatible JSONL.\n- A JSON summary output format for benchmark reports.\n- Custom response handler for accurate TTFT/ITL metrics with reasoning tokens (e.g., DeepSeek-R1).\n- Optional backend mode to preserve HTTP error details (`message/type/code`) in failed request records.\n\nInstall\n-------\nPython 3.10+ is required.\n\n```bash\npip install -e .\n```\n\nUsage\n-----\nShow available commands:\n\n```bash\nbenchmark-runner --help\n```\n\nRun a benchmark:\n\n```bash\nbenchmark-runner benchmark \\\n  --target http://localhost:8000 \\\n  --profile constant \\\n  --rate 10 \\\n  --max-seconds 20 \\\n  --data \"prompt_tokens=128,output_tokens=256\" \\\n  --processor PROCESSOR_PATH\n```\n\nProgress reporting\n------------------\nYou can send progress updates to a server endpoint during a benchmark:\n\n```bash\nbenchmark-runner benchmark \\\n  --target http://localhost:8000 \\\n  --profile constant \\\n  --rate 10 \\\n  --max-seconds 20 \\\n  --data \"prompt_tokens=128,output_tokens=256\" \\\n  --processor PROCESSOR_PATH \\\n  --progress-url https://example.com/api/progress/123 \\\n  --progress-auth YOUR_TOKEN\n```\n\nHTTP Error Details for Failed Requests\n--------------------------------------\nGuideLLM's default `openai_http` backend does not always preserve response-body\nerror payloads in request-level benchmark errors. Benchmark Runner provides an\nopt-in backend type that enriches failed request errors using OpenAI-style error\nfields (`error.message`, `error.type`, `error.code`):\n\n```bash\nbenchmark-runner benchmark run \\\n  --target http://localhost:8000/v1 \\\n  --backend openai_http_error_detail \\\n  --profile constant \\\n  --rate 10 \\\n  --max-requests 100 \\\n  --sample-requests 20 \\\n  --data \"prompt_tokens=128,output_tokens=256\" \\\n  --processor PROCESSOR_PATH\n```\n\nWhen a request fails, `requests.errored[*].info.error` in benchmark outputs will\ncontain text similar to:\n`HTTP 400: ... (type=BadRequestError, code=400)`.\n\nNote: if `--sample-requests 0` is used, request-level samples are omitted by design,\nincluding failed request details.\n\nShareGPT dataset support\n------------------------\nIf a dataset filename contains \"sharegpt\" and ends with `.json` or `.jsonl`,\nBenchmark Runner will convert it to a GuideLLM-compatible JSONL file before running\nthe benchmark.\n\nExample:\n\n```bash\nbenchmark-runner benchmark \\\n  --target http://localhost:8000 \\\n  --profile constant \\\n  --rate 10 \\\n  --max-seconds 20 \\\n  --processor PROCESSOR_PATH \\\n  --data ./ShareGPT_V3_unfiltered_cleaned_split.json\n```\n\nOutputs\n-------\nBenchmark Runner supports GuideLLM outputs plus a JSON summary output.\nTo save summary JSON:\n\n```bash\nbenchmark-runner benchmark \\\n  --target http://localhost:8000 \\\n  --profile constant \\\n  --rate 10 \\\n  --max-seconds 20 \\\n  --data \"prompt_tokens=128,output_tokens=256\" \\\n  --processor PROCESSOR_PATH \\\n  --outputs summary_json \\\n  --output-dir ./benchmarks\n```\n\nReasoning Tokens Support\n-------------------------\nFor models that output reasoning tokens (e.g., DeepSeek-R1, o1-preview), use the custom\nresponse handler to get accurate TTFT and ITL metrics:\n\n```bash\nbenchmark-runner benchmark run \\\n  --target http://localhost:8000/v1 \\\n  --backend openai_http \\\n  --backend-kwargs '{\"response_handlers\": {\"chat_completions\": \"chat_completions_with_reasoning\"}}' \\\n  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \\\n  --data your-dataset \\\n  --max-requests 100\n```\n\nDocker\n------\nThis repository includes a Dockerfile used to build a runtime image.\n\n```bash\ndocker build -t benchmark-runner .\n```\n\nDevelopment\n-----------\nInstall development dependencies:\n\n```bash\npip install -e \".[dev]\"\n```\n\nmacOS Notes\n-----------\nBenchmark Runner applies two macOS-only runtime defaults to avoid known\nmultiprocessing hangs:\n- switch GuideLLM multiprocessing context from `fork` to `spawn` (unless\n  `GUIDELLM__MP_CONTEXT_TYPE` is explicitly set)\n- default `--data-num-workers` to `0` unless provided on the CLI\n\nReferences:\n- https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods\n- https://bugs.python.org/issue33725\n\nTo disable these defaults for debugging/experiments:\n\n```bash\nBENCHMARK_RUNNER_DISABLE_MACOS_WORKAROUNDS=1 benchmark-runner benchmark run ...\n```\n\nLicense\n-------\nSee repository license information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpustack%2Fbenchmark-runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgpustack%2Fbenchmark-runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgpustack%2Fbenchmark-runner/lists"}