{"id":14081248,"url":"https://github.com/huggingface/optimum-benchmark","last_synced_at":"2025-10-14T15:33:16.634Z","repository":{"id":174021500,"uuid":"633052722","full_name":"huggingface/optimum-benchmark","owner":"huggingface","description":"🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations \u0026 quantization schemes.","archived":false,"fork":false,"pushed_at":"2025-09-24T07:07:21.000Z","size":9342,"stargazers_count":315,"open_issues_count":4,"forks_count":59,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-09-24T08:21:51.401Z","etag":null,"topics":["benchmark","neural-compressor","onnxruntime","openvino","pytorch","tensorrt-llm","text-generation-inference"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-04-26T17:19:02.000Z","updated_at":"2025-09-24T07:07:25.000Z","dependencies_parsed_at":"2023-09-23T02:26:13.429Z","dependency_job_id":"ed63e6a6-497c-4f71-b648-006af5d58179","html_url":"https://github.com/huggingface/optimum-benchmark","commit_stats":null,"previous_names":["huggingface/optimum-benchmark","ilyasmoutawwakil/optimum-benchmark"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/optimum-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/optimum-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019320,"owners_count":26086711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","neural-compressor","onnxruntime","openvino","pytorch","tensorrt-llm","text-generation-inference"],"created_at":"2024-08-13T13:00:35.439Z","updated_at":"2025-10-14T15:33:16.628Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Evaluation and Monitoring"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/huggingface/optimum-benchmark/main/logo.png\" alt=\"Optimum-Benchmark Logo\" width=\"350\" style=\"max-width: 100%;\" /\u003e\u003c/p\u003e\n\u003cp align=\"center\"\u003e\u003cq\u003eAll benchmarks are wrong, some will cost you less than others.\u003c/q\u003e\u003c/p\u003e\n\u003ch1 align=\"center\"\u003eOptimum-Benchmark 🏋️\u003c/h1\u003e\n\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/optimum-benchmark)](https://pypi.org/project/optimum-benchmark/)\n[![PyPI - Version](https://img.shields.io/pypi/v/optimum-benchmark)](https://pypi.org/project/optimum-benchmark/)\n[![PyPI - Implementation](https://img.shields.io/pypi/implementation/optimum-benchmark)](https://pypi.org/project/optimum-benchmark/)\n[![PyPI - Format](https://img.shields.io/pypi/format/optimum-benchmark)](https://pypi.org/project/optimum-benchmark/)\n[![PyPI - License](https://img.shields.io/pypi/l/optimum-benchmark)](https://pypi.org/project/optimum-benchmark/)\n\nOptimum-Benchmark is a unified [multi-backend \u0026 multi-device](#backends--devices-) utility for benchmarking [Transformers](https://github.com/huggingface/transformers), [Diffusers](https://github.com/huggingface/diffusers), [PEFT](https://github.com/huggingface/peft), [TIMM](https://github.com/huggingface/pytorch-image-models) and [Optimum](https://github.com/huggingface/optimum) libraries, along with all their supported [optimizations \u0026 quantization schemes](#backends--devices-), for [inference \u0026 training](#scenarios-), in [distributed \u0026 non-distributed settings](#launchers-), in the most correct, efficient and scalable way possible.\n\n*News* 📰\n\n- LlamaCpp backend for benchmarking [`llama-cpp-python`](https://github.com/abetlen/llama-cpp-python) bindings with all its supported devices 🚀\n- 🥳 PyPI package is now available for installation: `pip install optimum-benchmark` 🎉 [check it out](https://pypi.org/project/optimum-benchmark/) !\n- Model loading latency/memory/energy tracking for all backends in the inference scenario 🚀\n- numactl support for Process and Torchrun launchers to control the NUMA nodes on which the benchmark runs.\n- 4 minimal docker images (`cpu`, `cuda`, `rocm`) in [packages](https://github.com/huggingface/optimum-benchmark/pkgs/container/optimum-benchmark) for testing, benchmarking and reproducibility 🐳\n- vLLM backend for benchmarking [vLLM](https://github.com/vllm-project/vllm)'s inference engine 🚀\n- Hosting the codebase of the [LLM-Perf Leaderboard](https://huggingface.co/spaces/optimum/llm-perf-leaderboard) 🥇\n- Py-TXI backend for benchmarking [Py-TXI](https://github.com/IlyasMoutawwakil/py-txi/tree/main) 🚀\n- Python API for running isolated and distributed benchmarks with Python scripts 🐍\n- Simpler CLI interface for running benchmarks (runs and sweeps) using the Hydra 🧪\n\n*Motivations* 🎯\n\n- HuggingFace hardware partners wanting to know how their hardware performs compared to another hardware on the same models.\n- HuggingFace ecosystem users wanting to know how their chosen model performs in terms of latency, throughput, memory usage, energy consumption, etc compared to another model.\n- Benchmarking hardware \u0026 backend specific optimizations \u0026 quantization schemes that can be applied to models and improve their computational/memory/energy efficiency.\n\n\u0026#160;\n\u003e \\[!Note\\]\n\u003e Optimum-Benchmark is a work in progress and is not yet ready for production use, but we're working hard to make it so. Please keep an eye on the project and help us improve it and make it more useful for the community. We're looking forward to your feedback and contributions. 🚀\n\u0026#160;\n\n## CI Status 🚦\n\nOptimum-Benchmark is continuously and intensively tested on a variety of devices, backends, scenarios and launchers to ensure its stability with over 300 tests running on every PR (you can request more tests if you want to).\n\n### API 📈\n\n[![API_CPU](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_cpu.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_cpu.yaml)\n[![API_CUDA](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_cuda.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_cuda.yaml)\n[![API_MISC](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_misc.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_misc.yaml)\n[![API_ROCM](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_rocm.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_api_rocm.yaml)\n\n### CLI 📈\n\n[![CLI_CPU_IPEX](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_ipex.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_ipex.yaml)\n[![CLI_CPU_LLAMA_CPP](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_llama_cpp.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_llama_cpp.yaml)\n[![CLI_CPU_ONNXRUNTIME](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_onnxruntime.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_onnxruntime.yaml)\n[![CLI_CPU_OPENVINO](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_openvino.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_openvino.yaml)\n[![CLI_CPU_PYTORCH](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_pytorch.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_pytorch.yaml)\n[![CLI_CPU_PY_TXI](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_py_txi.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cpu_py_txi.yaml)\n[![CLI_CUDA_ONNXRUNTIME](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_onnxruntime.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_onnxruntime.yaml)\n[![CLI_CUDA_PYTORCH](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_pytorch.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_pytorch.yaml)\n[![CLI_CUDA_PY_TXI](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_py_txi.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_py_txi.yaml)\n[![CLI_CUDA_TENSORRT_LLM](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_tensorrt_llm.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_tensorrt_llm.yaml)\n[![CLI_CUDA_VLLM](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_vllm.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_cuda_vllm.yaml)\n[![CLI_MISC](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_misc.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_misc.yaml)\n[![CLI_ROCM_PYTORCH](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_rocm_pytorch.yaml/badge.svg)](https://github.com/huggingface/optimum-benchmark/actions/workflows/test_cli_rocm_pytorch.yaml)\n\n## Quickstart 🚀\n\n### Installation 📥\n\n#### Using uv (Recommended)\n\n[uv](https://docs.astral.sh/uv/) is a fast Python package manager that we recommend for installing and managing dependencies:\n\n```bash\n# Install uv if you haven't already\npip install uv\n\n# Add optimum-benchmark to your uv project\nuv add optimum-benchmark\n\n# Or run optimum-benchmark with uv as a command without installing it \n# (automaticlly installs and runs it in an isolated virtual environment)\nuv run optimum-benchmark --config-dir examples/ --config-name cuda_pytorch_bert\n\n# Or run optimum-benchmark with uv script without installing it\n# (automaticlly installs and runs it in an isolated virtual environment)\nuv run script.py\nscript.py:\n# /// script\n# dependencies = [\n#   \"optimum-benchmark\",\n# ]\n# ///\n# [...]\n\n# Or clone the repository and create an environment with optimum-benchmark installed for development\ngit clone https://github.com/huggingface/optimum-benchmark.git\ncd optimum-benchmark\nuv sync\n```\n\n#### Using pip\n\nYou can also install using traditional pip:\n\n```bash\npip install optimum-benchmark\n```\n\nor install the latest version from the main branch:\n\n```bash\npip install git+https://github.com/huggingface/optimum-benchmark.git\n```\n\nor if you want to tinker with the code, you can clone the repository and install it in editable mode:\n\n```bash\ngit clone https://github.com/huggingface/optimum-benchmark.git\ncd optimum-benchmark\npip install -e .\n```\n\n\u003cdetails\u003e\n    \u003csummary\u003eAdvanced install options\u003c/summary\u003e\n\nDepending on the backends you want to use, you can install `optimum-benchmark` with the following extras:\n\n#### With uv (Recommended):\n- PyTorch (default): `uv add optimum-benchmark`\n- OpenVINO: `uv add optimum-benchmark --extra openvino`\n- ONNXRuntime: `uv add optimum-benchmark --extra onnxruntime`\n- TensorRT-LLM: `uv add optimum-benchmark --extra tensorrt-llm`\n- ONNXRuntime-GPU: `uv add optimum-benchmark --extra onnxruntime-gpu`\n- Py-TXI (TGI \u0026 TEI): `uv add optimum-benchmark --extra py-txi`\n- vLLM: `uv add optimum-benchmark --extra vllm`\n- IPEX: `uv add optimum-benchmark --extra ipex`\n\n#### With pip:\n- PyTorch (default): `pip install optimum-benchmark`\n- OpenVINO: `pip install optimum-benchmark[openvino]`\n- ONNXRuntime: `pip install optimum-benchmark[onnxruntime]`\n- TensorRT-LLM: `pip install optimum-benchmark[tensorrt-llm]`\n- ONNXRuntime-GPU: `pip install optimum-benchmark[onnxruntime-gpu]`\n- Py-TXI (TGI \u0026 TEI): `pip install optimum-benchmark[py-txi]`\n- vLLM: `pip install optimum-benchmark[vllm]`\n- IPEX: `pip install optimum-benchmark[ipex]`\n\nWe also support the following extra dependencies:\n\n- gptqmodel\n- sentence-transformers\n- bitsandbytes\n- codecarbon\n- flash-attn\n- deepspeed\n- diffusers\n- timm\n- peft\n\n\u003c/details\u003e\n\n### Running benchmarks using the Python API 🧪\n\nYou can run benchmarks from the Python API as simple python scripts. This is especially cool when used with [`uv` scripts](https://docs.astral.sh/uv/guides/scripts/).\nHere's an example of how to run an isolated benchmark using the `pytorch` backend, `torchrun` launcher and `inference` scenario with latency and memory tracking enabled.\n\n```python\n# /// script\n# dependencies = [\n#   \"optimum-benchmark\",\n# ]\n# ///\n\nfrom optimum_benchmark import Benchmark, BenchmarkConfig, TorchrunConfig, InferenceConfig, PyTorchConfig\nfrom optimum_benchmark.logging_utils import setup_logging\n\nsetup_logging(level=\"INFO\")\n\nif __name__ == \"__main__\":\n    launcher_config = TorchrunConfig(nproc_per_node=2)\n    scenario_config = InferenceConfig(latency=True, memory=True)\n    backend_config = PyTorchConfig(model=\"gpt2\", device=\"cuda\", device_ids=\"0,1\", no_weights=True)\n    benchmark_config = BenchmarkConfig(\n        name=\"pytorch_gpt2\",\n        scenario=scenario_config,\n        launcher=launcher_config,\n        backend=backend_config,\n    )\n    benchmark_report = Benchmark.launch(benchmark_config)\n    # push artifacts to the hub\n    benchmark_config.push_to_hub(\"IlyasMoutawwakil/pytorch_gpt2\") # or benchmark_config.save_json(\"pytorch_gpt2/benchmark_config.json\")\n    benchmark_report.push_to_hub(\"IlyasMoutawwakil/pytorch_gpt2\") # or benchmark_report.save_json(\"pytorch_gpt2/benchmark_report.json\")\n\n    # merge them into a single artifact and push to the hub\n    benchmark = Benchmark(config=benchmark_config, report=benchmark_report)\n    benchmark.push_to_hub(\"IlyasMoutawwakil/pytorch_gpt2\") # or benchmark.save_json(\"pytorch_gpt2/benchmark.json\")\n```\n\nRunning the above with `uv` will automatically runs it in an isolated virtual environment with `optimum-benchmark` installed, and it's as simple as:\n\n```bash\nuv run script.py\n```\n\nYou can also see the available parameters of different configuration classes in the [Features](#features-) section below.\n\n### Running benchmarks using the Hydra CLI 🧪\n\nYou can also run a benchmark using the command line by specifying the configuration directory and the configuration name. Both arguments are mandatory for [`hydra`](https://hydra.cc/). `--config-dir` is the directory where the configuration files are stored and `--config-name` is the name of the configuration file without its `.yaml` extension.\n\n```bash\noptimum-benchmark --config-dir examples/ --config-name cuda_pytorch_bert\n```\n\nThis will run the benchmark using the configuration in [`examples/cuda_pytorch_bert.yaml`](examples/cuda_pytorch_bert.yaml) and store the results in `runs/cuda_pytorch_bert`.\n\nThe resulting files are :\n\n- `benchmark_config.json` which contains the configuration used for the benchmark, including the backend, launcher, scenario and the environment in which the benchmark was run.\n- `benchmark_report.json` which contains a full report of the benchmark's results, like latency measurements, memory usage, energy consumption, etc.\n- `benchmark_report.txt` which contains a detailed report of the benchmark's results, in the same format they were logged.\n- `benchmark_report.md` which contains a detailed report of the benchmark's results, in markdown format.\n- `benchmark.json` contains both the report and the configuration in a single file.\n- `benchmark.log` contains the logs of the benchmark run.\n\n\u003cdetails\u003e\n\u003csummary\u003eAdvanced CLI options\u003c/summary\u003e\n\n#### Configuration overrides 🎛️\n\nIt's easy to override the default behavior of a benchmark from the command line of an already existing configuration file. For example, to run the same benchmark on a different device, you can use the following command:\n\n```bash\noptimum-benchmark --config-dir examples/ --config-name pytorch_bert backend.model=gpt2 backend.device=cuda\n```\n\n#### Configuration sweeps 🧹\n\nYou can easily run configuration sweeps using the `--multirun` option. By default, configurations will be executed serially but other kinds of executions are supported with hydra's launcher plugins (e.g. `hydra/launcher=joblib`).\n\n```bash\noptimum-benchmark --config-dir examples --config-name pytorch_bert -m backend.device=cpu,cuda\n```\n\n### Configurations structure 📁\n\nYou can create custom and more complex configuration files following these [examples]([examples](https://github.com/IlyasMoutawwakil/optimum-benchmark-examples)). They are heavily commented to help you understand the structure of the configuration files.\n\n\u003c/details\u003e\n\n## Features 🎨\n\n`optimum-benchmark` allows you to run benchmarks with minimal configuration. A benchmark is defined by three main components:\n\n- The launcher to use (e.g. `process`)\n- The scenario to follow (e.g. `training`)\n- The backend to run on (e.g. `onnxruntime`)\n\n### Launchers 🚀\n\n- [x] Process launcher (`launcher=process`); Launches the benchmark in an isolated process.\n- [x] Torchrun launcher (`launcher=torchrun`); Launches the benchmark in multiples processes using `torch.distributed`.\n- [x] Inline launcher (`launcher=inline`), not recommended for benchmarking, only for debugging purposes.\n\n\u003cdetails\u003e\n\u003csummary\u003eGeneral Launcher features 🧰\u003c/summary\u003e\n\n- [x] Assert GPU devices (NVIDIA \u0026 AMD) isolation (`launcher.device_isolation=true`). This feature makes sure no other processes are running on the targeted GPU devices other than the benchmark. Espepecially useful when running benchmarks on shared resources.\n\n\u003c/details\u003e\n\n### Scenarios 🏋\n\n- [x] Training scenario (`scenario=training`) which benchmarks the model using the trainer class with a randomly generated dataset.\n- [x] Inference scenario (`scenario=inference`) which benchmakrs the model's inference method (forward/call/generate) with randomly generated inputs.\n\n\u003cdetails\u003e\n\u003csummary\u003eInference scenario features 🧰\u003c/summary\u003e\n\n- [x] Memory tracking (`scenario.memory=true`)\n- [x] Energy and efficiency tracking (`scenario.energy=true`)\n- [x] Latency and throughput tracking (`scenario.latency=true`)\n- [x] Warm up runs before inference (`scenario.warmup_runs=20`)\n- [x] Inputs shapes control (e.g. `scenario.input_shapes.sequence_length=128`)\n- [x] Forward, Call and Generate kwargs (e.g. for an LLM `scenario.generate_kwargs.max_new_tokens=100`, for a diffusion model `scenario.call_kwargs.num_images_per_prompt=4`)\n\nSee [InferenceConfig](optimum_benchmark/scenarios/inference/config.py) for more information.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eTraining scenario features 🧰\u003c/summary\u003e\n\n- [x] Memory tracking (`scenario.memory=true`)\n- [x] Energy and efficiency tracking (`scenario.energy=true`)\n- [x] Latency and throughput tracking (`scenario.latency=true`)\n- [x] Warm up steps before training (`scenario.warmup_steps=20`)\n- [x] Dataset shapes control (e.g. `scenario.dataset_shapes.sequence_length=128`)\n- [x] Training arguments control (e.g. `scenario.training_args.per_device_train_batch_size=4`)\n\nSee [TrainingConfig](optimum_benchmark/scenarios/training/config.py) for more information.\n\n\u003c/details\u003e\n\n### Backends \u0026 Devices 📱\n\n- [x] PyTorch backend for CPU (`backend=pytorch`, `backend.device=cpu`)\n- [x] PyTorch backend for CUDA (`backend=pytorch`, `backend.device=cuda`, `backend.device_ids=0,1`)\n- [ ] PyTorch backend for Habana Gaudi Processor (`backend=pytorch`, `backend.device=hpu`, `backend.device_ids=0,1`)\n- [x] ONNXRuntime backend for CPUExecutionProvider (`backend=onnxruntime`, `backend.device=cpu`)\n- [x] ONNXRuntime backend for CUDAExecutionProvider (`backend=onnxruntime`, `backend.device=cuda`)\n- [x] ONNXRuntime backend for ROCMExecutionProvider (`backend=onnxruntime`, `backend.device=cuda`, `backend.provider=ROCMExecutionProvider`)\n- [x] ONNXRuntime backend for TensorrtExecutionProvider (`backend=onnxruntime`, `backend.device=cuda`, `backend.provider=TensorrtExecutionProvider`)\n- [x] Py-TXI backend for CPU and GPU (`backend=py-txi`, `backend.device=cpu` or `backend.device=cuda`)\n- [x] Neural Compressor backend for CPU (`backend=neural-compressor`, `backend.device=cpu`)\n- [x] TensorRT-LLM backend for CUDA (`backend=tensorrt-llm`, `backend.device=cuda`)\n- [x] OpenVINO backend for CPU (`backend=openvino`, `backend.device=cpu`)\n- [x] OpenVINO backend for GPU (`backend=openvino`, `backend.device=gpu`)\n- [x] vLLM backend for CUDA (`backend=vllm`, `backend.device=cuda`)\n- [x] vLLM backend for ROCM (`backend=vllm`, `backend.device=rocm`)\n- [x] vLLM backend for CPU (`backend=vllm`, `backend.device=cpu`)\n- [x] IPEX backend for CPU (`backend=ipex`, `backend.device=cpu`)\n- [x] IPEX backend for XPU (`backend=ipex`, `backend.device=xpu`)\n\n\u003cdetails\u003e\n\u003csummary\u003eGeneral backend features 🧰\u003c/summary\u003e\n\n- [x] Device selection (`backend.device=cuda`), can be `cpu`, `cuda`, `mps`, etc.\n- [x] Device ids selection (`backend.device_ids=0,1`), can be a list of device ids to run the benchmark on multiple devices.\n- [x] Model selection (`backend.model=gpt2`), can be a model id from the HuggingFace model hub or an **absolute path** to a model folder.\n- [x] \"No weights\" feature, to benchmark models without downloading their weights, using randomly initialized weights (`backend.no_weights=true`)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eBackend specific features 🧰\u003c/summary\u003e\n\nFor more information on the features of each backend, you can check their respective configuration files:\n\n- [VLLMConfig](optimum_benchmark/backends/vllm/config.py)\n- [IPEXConfig](optimum_benchmark/backends/ipex/config.py)\n- [OpenVINOConfig](optimum_benchmark/backends/openvino/config.py)\n- [PyTXIConfig](optimum_benchmark/backends/py_txi/config.py)\n- [PyTorchConfig](optimum_benchmark/backends/pytorch/config.py)\n- [ONNXRuntimeConfig](optimum_benchmark/backends/onnxruntime/config.py)\n- [TRTLLMConfig](optimum_benchmark/backends/tensorrt_llm/config.py)\n\n\u003c/details\u003e\n\n## Contributing 🤝\n\nContributions are welcome! And we're happy to help you get started. Feel free to open an issue or a pull request.\nThings that we'd like to see:\n\n- More backends (Tensorflow, TFLite, Jax, etc).\n- More tests (for optimizations and quantization schemes).\n- More hardware support (Habana Gaudi Processor (HPU), Apple M series, etc).\n- Task evaluators for the most common tasks (would be great for output regression).\n\nTo get started, you can check the [CONTRIBUTING.md](CONTRIBUTING.md) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Foptimum-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-benchmark/lists"}