{"id":27739745,"url":"https://github.com/kubernetes-sigs/inference-perf","last_synced_at":"2025-08-07T23:54:55.803Z","repository":{"id":272517836,"uuid":"916860795","full_name":"kubernetes-sigs/inference-perf","owner":"kubernetes-sigs","description":"GenAI inference performance benchmarking tool","archived":false,"fork":false,"pushed_at":"2025-03-31T22:12:39.000Z","size":218,"stargazers_count":39,"open_issues_count":20,"forks_count":11,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-04-21T06:55:26.002Z","etag":null,"topics":["k8s-sig-scalability"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kubernetes-sigs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-14T22:26:31.000Z","updated_at":"2025-04-19T04:20:59.000Z","dependencies_parsed_at":"2025-03-20T17:41:43.614Z","dependency_job_id":null,"html_url":"https://github.com/kubernetes-sigs/inference-perf","commit_stats":null,"previous_names":["kubernetes-sigs/inference-perf"],"tags_count":0,"template":false,"template_full_name":"kubernetes/kubernetes-template-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Finference-perf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Finference-perf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Finference-perf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Finference-perf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kubernetes-sigs","download_url":"https://codeload.github.com/kubernetes-sigs/inference-perf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251342721,"owners_count":21574244,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k8s-sig-scalability"],"created_at":"2025-04-28T16:00:43.291Z","updated_at":"2025-08-07T23:54:55.743Z","avatar_url":"https://github.com/kubernetes-sigs.png","language":"Python","funding_links":[],"categories":["Inference"],"sub_categories":["Benchmark"],"readme":"# Inference Perf\n\nInference Perf is a GenAI inference performance benchmarking tool. It came out of [wg-serving](https://github.com/kubernetes/community/tree/master/wg-serving) and is sponsored by [SIG Scalability](https://github.com/kubernetes/community/blob/master/sig-scalability/README.md#inference-perf). Original proposal can be found [here](https://github.com/kubernetes-sigs/wg-serving/tree/main/proposals/013-inference-perf).\n\n## Architecture\n\n![Architecture Diagram](docs/images/architecture.png)\n\n## Key Features\n\n* Highly scalable and can support benchmarking large inference production deployments.\n* Reports the key metrics needed to measure LLM performance.\n* Supports different real world and synthetic datasets.\n* Supports different APIs and can support multiple model servers.\n* Supports specifying an exact input and output distribution to simulate different scenarios - Gaussian distribution, fixed length, min-max cases are all supported.\n* Generates different load patterns and can benchmark specific cases like burst traffic, scaling to saturation and other autoscaling / routing scenarios.\n\n## Roadmap\n\n* Accelerator metrics collection during benchmarks (GPU utilization, memory usage, power usage, etc.).\n* Support for more model servers.\n* Deployment API to help deploy different inference stacks.\n* Traffic splitting among different use cases or LoRA adapters.\n* Support for benchmarking non-LLM GenAI use cases.\n* Support for different datasets to simulate real world use cases.\n* Replaying traffic from production systems.\n\n## Getting Started\n\n### Run locally\n\n- Setup a virtual environment and install inference-perf\n\n    ```\n    pip install inference-perf\n    ```\n\n- Run inference-perf CLI with a configuration file\n\n    ```\n    inference-perf --config_file config.yml\n    ```\n\n- See more [examples](./examples/)\n\n### Run in a Docker container\n\n- Run the container by mounting your config file.\n\n    ```\n    docker run -it --rm -v $(pwd)/config.yml:/workspace/config.yml quay.io/inference-perf/inference-perf\n    ```\n\n### Run in a Kubernetes cluster\n\nRefer to the [guide](./deploy/README.md) in `/deploy`.\n\n## Configuration\n\nYou can configure inference-perf to run with different data generation and load generation configurations today. Please see `config.yml` and examples in `/examples`.\n\nRefer to the [CONFIG.md](./CONFIG.md) for documentation on all supported configuration options.\n\n### Datasets\n\nSupported datasets include the following:\n- [ShareGPT](./examples/vllm/config.yml) (for a real world conversational dataset)\n- [Synthetic](./examples/vllm/config-synthetic.yml) (for specific input / output distributions with Sonnet data)\n- [Random](./examples/vllm/config-random.yml) (for specific input / output distributions with random data)\n- [SharedPrefix](./examples/vllm/config-shared-prefix.yml) (for prefix caching scenarios)\n- mock (for testing)\n\n### Load Generators\n\nMultiple load generators are supported:\n- Poisson / constant-time load generation to send specific QPS.\n- Multi-process load generation for increased concurrency and higher QPS.\n\nMultiple load patterns can be specified:\n- Stages with configurable duration and QPS along with specific timeouts in between them allows you to simulate different load patterns like burst in traffic, constantly increasing load till hardware saturation, etc.\n\n### API\n\nOpenAI completion and chat completion APIs are supported. It can be pointed to any endpoints which support these APIs - currently verified against vLLM deployments. Other APIs and model server support can be added easily.\n\n### Metrics\n\nDifferent latency and throughput metrics to analyze the performance of different LLM workloads are reported. A snippet from an example report is below.\n```\n\"latency\": {\n    \"request_latency\": {\n        \"mean\": 3.31325431142327,\n        \"min\": 1.62129471905064,\n        \"p10\": 1.67609986825846,\n        \"p50\": 2.11507539497688,\n        \"p90\": 5.94717199734878,\n        \"max\": 6.30658466403838\n    },\n    \"normalized_time_per_output_token\": {\n        \"mean\": 0.104340420636009,\n        \"min\": 0.0506654599703325,\n        \"p10\": 0.0523781208830769,\n        \"p50\": 0.0670631669655753,\n        \"p90\": 0.189047570470012,\n        \"max\": 0.20343821496898\n    },\n    \"time_per_output_token\": {\n        \"mean\": 0.0836929455635872,\n        \"min\": 0.0517028436646797,\n        \"p10\": 0.0530815053513894,\n        \"p50\": 0.0611870964678625,\n        \"p90\": 0.152292036800645,\n        \"max\": 0.17837208439984\n    },\n    \"time_to_first_token\": {\n        \"mean\": 0.800974442732916,\n        \"min\": 0.0625283779809251,\n        \"p10\": 0.072068731742911,\n        \"p50\": 0.203539535985328,\n        \"p90\": 2.26959549135063,\n        \"max\": 4.46773961000145\n    },\n    \"inter_token_latency\": {\n        \"mean\": 0.0836929455635872,\n        \"min\": 0.000007129972800612,\n        \"p10\": 0.0534287681337446,\n        \"p50\": 0.0591336835059337,\n        \"p90\": 0.084046097996179,\n        \"max\": 0.614475268055685\n    }\n},\n\"throughput\": {\n    \"input_tokens_per_sec\": 643.576644186323,\n    \"output_tokens_per_sec\": 32.544923821416,\n    \"total_tokens_per_sec\": 676.121568007739,\n    \"requests_per_sec\": 1.0238155253639\n},\n\"prompt_len\": {\n    \"mean\": 628.606060606061,\n    \"min\": 4,\n    \"p10\": 11.4,\n    \"p50\": 364,\n    \"p90\": 2427.6,\n    \"max\": 3836\n},\n\"output_len\": {\n    \"mean\": 31.7878787878788,\n    \"min\": 30,\n    \"p10\": 31,\n    \"p50\": 32,\n    \"p90\": 32,\n    \"max\": 32\n}\n```\n\n### Reports\n\nReports are generated in JSON format.\n- Per stage reports for individual request rates.\n- Summary reports for the overall run.\n- Request logs / traces for further analysis.\n\nModel server metrics reports from Prometheus collected during the run is also produced.\n- Model server specific metrics like queue size, batch size, latency metrics, etc.\n- Supports querying metrics from OSS Prometheus and Google Managed Prometheus.\n\n## Analysis\n\nReports can be analyzed using the following command:\n\n```\ninference-perf --analyze \u003cpath-to-dir-with-reports\u003e\n```\n\nThis should generate the following charts (below charts are for example only):\n\n1. QPS vs Latency (TTFT, NTPOT, ITL)\n\n![qps-latency-chart](./docs/images/latency_vs_qps.png)\n\n2. QPS vs Throughput (input tokens / sec, output tokens / sec, total tokens / sec)\n\n![qps-throughput-chart](./docs/images/throughput_vs_qps.png)\n\n3. Latency vs Throughput (output tokens / sec vs TTFT, NTPOT and ITL)\n\n![latency-throughput-chart](./docs/images/throughput_vs_latency.png)\n\n## Contributing\n\nOur community meeting is weekly on Thursdays alternating betweem 09:00 and 11:30 PDT ([Zoom Link](https://zoom.us/j/9955436256?pwd=Z2FQWU1jeDZkVC9RRTN4TlZyZTBHZz09), [Meeting Notes](https://docs.google.com/document/d/15XSF8q4DShcXIiExDfyiXxAYQslCmOmO2ARSJErVTak/edit?usp=sharing), [Meeting Recordings](https://www.youtube.com/playlist?list=PL69nYSiGNLP30qNanabU75ayPK7OPNAAS)). \n\nWe currently utilize the [#inference-perf](https://kubernetes.slack.com/?redir=%2Fmessages%2Finference-perf) channel in Kubernetes Slack workspace for communications.\n\nContributions are welcomed, thanks for joining us!\n\n### Code of conduct\n\nParticipation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubernetes-sigs%2Finference-perf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkubernetes-sigs%2Finference-perf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubernetes-sigs%2Finference-perf/lists"}