{"id":28529449,"url":"https://github.com/mlcommons/mlperf_client","last_synced_at":"2026-04-20T21:02:09.727Z","repository":{"id":290387859,"uuid":"901579283","full_name":"mlcommons/mlperf_client","owner":"mlcommons","description":"MLPerf Client is a benchmark for Windows, Linux and macOS, focusing on client form factors in ML inference scenarios.","archived":false,"fork":false,"pushed_at":"2026-04-10T18:54:23.000Z","size":82427,"stargazers_count":82,"open_issues_count":7,"forks_count":6,"subscribers_count":14,"default_branch":"main","last_synced_at":"2026-04-10T20:27:00.918Z","etag":null,"topics":["benchmarking","machine-learning","ml"],"latest_commit_sha":null,"homepage":"https://mlcommons.org/benchmarks/client/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlcommons.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-12-10T23:29:41.000Z","updated_at":"2026-04-10T18:54:26.000Z","dependencies_parsed_at":"2025-04-28T15:50:07.088Z","dependency_job_id":null,"html_url":"https://github.com/mlcommons/mlperf_client","commit_stats":null,"previous_names":["mlcommons/mlperf_client"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/mlcommons/mlperf_client","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmlperf_client","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmlperf_client/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmlperf_client/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmlperf_client/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlcommons","download_url":"https://codeload.github.com/mlcommons/mlperf_client/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlcommons%2Fmlperf_client/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32065584,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-20T11:35:06.609Z","status":"ssl_error","status_checked_at":"2026-04-20T11:34:48.899Z","response_time":94,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","machine-learning","ml"],"created_at":"2025-06-09T14:10:54.250Z","updated_at":"2026-04-20T21:02:09.678Z","avatar_url":"https://github.com/mlcommons.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MLPerf Client benchmark\n\nMLPerf Client is a benchmark for Windows and macOS, focusing on client form factors in ML inference scenarios like AI chatbots,\nimage classification, etc. The benchmark evaluates performance across different hardware and software configurations,\nproviding command line interface.\n\n## Table of Contents\n- [Features](#features)\n- [Running the Application](#running-the-application)\n - [Windows Users](#windows-users)\n- [Command Line Arguments](#command-line-arguments)\n - [Arguments Reference](#arguments-reference)\n - [Usage Examples](#usage-examples)\n - [Arguments Behavior Summary](#arguments-behavior-summary)\n- [Configuration File Format](#configuration-file-format)\n - [URIs Supported](#uris-supported)\n - [Tested Workflows](#tested-workflows)\n- [Supported Execution Providers and Configurations](#supported-execution-providers-and-configurations)\n - [Status Definitions](#status-definitions)\n - [Notes](#notes)\n- [Supported Platforms by Execution Provider](#supported-platforms-by-execution-provider)\n\n\n\n## Features\n\nThe MLPerf Client benchmark measures the performance of inference tasks on personal computers. It supports the following features:\n\n- The application provides reference models for inference, including support for models such as Llama 2.\n- Independent Hardware Vendors {IHVs} offer paths as Execution Providers, with their own custom model format and stack dependencies, which adhere to IHV APIs.\n- The client bundles all enabled/active paths from the EP list (see below)\n- Configure hardware and inference parameters using a JSON configuration file.\n- The final build product is a single binary for Windows and MacOS, with model and data files downloaded as needed.\n\n## Running the Application\n\n### Windows Users\n\nYou can run the application using:\n\n```cmd\n.\\mlperf-windows.exe [options]\n```\n\n## Command Line Arguments\n\n### Arguments Reference\n\nBelow are the available command line arguments that control the tool's behavior:\n\n| Argument | Description | Required | Default Value | Available Values |\n|------------------------------|------------------------------------------------------------------------------|----------|---------------|-----------------------------|\n| `-h, --help` | Show help message and exit | No | | |\n| `-v, --version` | Show the version of the tool | No | | |\n| `-c, --config` | Path to the configuration file | Yes | | Any valid path |\n| `-o, --output-dir` | Specify output directory. If not provided, default output directory is used. | No |\n| `-d, --data-dir` | Specify directory, where all required data files will be downloaded | No | `data` |\n| `-p, --pause` | Flag to allow pausing the program at the end of execution. | No | `true` | `true`, `false` |\n| `-l, --logger` | Path to the log4cxx configuration file | No | `log4cxx.xml` | Any valid path |\n| `-m, --list-models` | List all the available models supported by the tool | No | | |\n| `-b, --download_behaviour` | Controls The files download behaviour | No | `normal` | `forced`, `prompt`, `skip_all`, `deps_only `, `normal` |\n| `-n, --cache-local-files` | If it is set to false local files specified in the config will not be copied and cached | No | `true` | `true`, `false`|\n| `-i, --device-id` | Overrides device id in the config file for EP supporting devie enumeration. Use 'All' to run benchmark on all devices sequentially | No | | `all`, `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `9` |\n| `-e, --enumerate-devices` | Enumerates available devices for all EPs present in the provided config file and exits | No | `true` | `true`, `false`|\n\n\n### Usage Examples\n\nTypical usage of the tool would look like this:\n\n```bash\n.\\mlperf-windows.exe -c NVIDIA_ORTGenAI-DML_GPU.json\n```\n\nThis command runs the benchmark using `NVIDIA_ORTGenAI-DML_GPU.json` as the configuration file.\n\n### Arguments Behavior Summary\n\n- `-b` Controls the behavior of the application's file downloading process.\n - `forced`: The application will download the necessary files even if they are already present.\n - `prompt`: The application will check for the existence of required files.\n If any are missing, the user will be\n prompted to decide whether to download the missing files or abort the operation.\n - `skip_all`: The application will skip downloading any files.\n If a required file is missing, an error will be displayed, \n and the operation will be aborted.\n - `deps_only`: The application will download only the required dependencies and exit.\n - `normal` (default): The application will check the local cache first. If files are not cached or their URIs have changed, it will download the missing/updated files and cache them for future use.\n\n\nNow, let's see how to create a configuration file for the tool.\n\n## Configuration File Format\n\n*Before reading this section, please check the repository's data folder for sample configuration files.*\nThe configuration file is a JSON file that stores the tool's settings. This is an example of a configuration file.\n\n```json\n{\n \"SystemConfig\": {\n \"Comment\": \"Default config\",\n \"TempPath\": \"\"\n },\n \"Scenarios\": [\n list\n of\n scenarios\n here\n ]\n}\n```\n\nWhere:\n\n- SystemConfig: Contains the system configuration settings for the tool.\n - `Comment`: A comment for the configuration file that describes the configuration.\n - `TempPath`: The path to the temporary directory where the tool will store the downloaded files. If this is not\n provided, the tool will use the system's temporary directory.\n - `BaseDir`: The `BaseDir` is an Optional field that can be used to specify a base directory that will be prepended\n to all relative paths defined in the configuration file. When a relative path is detected in the configuration,\n the tool automatically prepends the basedir value to it.\n - `DownloadBehavior`: The `DownloadBehavior` is an Optional field that do exactly what the `-b` command line argument does.\n It controls the behavior of the application's file downloading process. If the `DownloadBehavior` provided in both the\n configuration file and the command line, the command line argument will take precedence.\n - `CacheLocalFiles`: The `CacheLocalFiles` is an Optional field that do exactly what the `--cache-local-files` command line argument does. If it is set to false local files specified in the config will not be copied and cached. If the `CacheLocalFiles` provided in both the configuration file and the command line, the command line argument will take precedence\n\n- Scenarios: A list of scenarios to run using the tool. Each scenario has the following format:\n ```json\n {\n \"Name\": \"Llama2\",\n \"Models\": [list of models here],\n \"InputFilePath\": [ \"input file path1\", \"input file path2\", ...],\n \"AssetsPath\": [ \"assets path1\", \"assets path2\", ...],\n \"ResultsVerificationFile\": \"path to results verification file\",\n \"DataVerificationFile\": \"path to data verification file\",\n \"Iterations\": 1000,\n \"WarmUp\": 1,\n \"Delay\": 0,\n \"ExecutionProviders\": [list of execution providers here]\n }\n ```\n Where:\n - `Name`: the name of the scenario.\n - `Models`: a list of models to use for the scenario, the tool will run the scenario using each of the models in the\n list.\n each model has the following format:\n ```json\n {\n \"ModelName\": \"Llama2 llama-2-7b-chat-dml\",\n \"FilePath\": \"https://client.mlcommons-storage.org/deps/0.6/scenario_files/llm/llama2/models/OrtGenAI/llama-2-7b-chat-dml/model.onnx\",\n \"DataFilePath\": \"https://client.mlcommons-storage.org/deps/0.6/scenario_files/llm/llama2/models/OrtGenAI/llama-2-7b-chat-dml/model.onnx.data.zip\",\n \"TokenizerPath\": \"https://client.mlcommons-storage.org/deps/0.6/scenario_files/llm/llama2/models/OrtGenAI/llama-2-7b-chat-dml/tokenizer.zip\"\n }\n ```\n Where:\n - `ModelName`: Specifies the model name.\n - `FilePath`: Specifies the path to the main model file (e.g., llama2-cpu-int4/model.onnx).\n - `DataFilePath`: Specifies the path to the data file associated with the model (e.g.,\n llama-2-7b-chat-dml/model.onnx.data). This file is required for executing the Llama2 scenario. Some models might have this file splitted into multiple\n files. To avoid setting manually all the files in this dictionary, these can be zipped into a single\n file - the app will take care of the unzipping.\n - `TokenizerPath`: Specifies the path to the tokenizer file (e.g., llama-2-7b-chat/tokenizer.zip). This file is\n required for executing the Llama2 scenario and contains the necessary tokenizer configuration and data.(The\n file does not need to be in ZIP format; it can be any supported format compatible with the model.)\n\n - `InputFilePath`: is a list of paths to the data files used in the scenario (input data) this could be a list of\n json files or even zip files.\n - `AssetsPath`: is a list of paths to the assets files used in the scenario.\n - `ResultsVerificationFile`: is the path to the file that contains the expected results of the scenario, ie the\n excepted output for each input.\n - `DataVerificationFile`: a path to the file that contains the SHA256 hash of all the files used in the scenario,\n this is used to verify the integrity of the data files. Please note that this file is optional, but if it's\n provided, it should contain the hashes of all the files used in the scenario.\n The format of the `DataVerificationFile` should be:\n ```json\n {\n \"FileHashes\": [\n \"8b13ac3ccb407ecbbcae0fe427f065c44dd96e259935de35699e607ec5ba12bc\",\n ...\n \n ]\n }\n ```\n\n The order of the hashes shouldn't matter, the tool will verify the integrity of the files by comparing the hashes\n of the files with the hashes in the `DataVerificationFile`.\n\n - `Iterations`: the number of iterations to run the scenario for.\n - `WarmUp`: the number of warmup iterations to run the scenario for.\n - `Delay`: the delay in seconds between each scenario iteration. This is an optional argument, and the default\n values varies by model. For `llama2`.\n - `ExecutionProviders`: a list of execution providers to use for the scenario, the tool will run the scenario using\n each of the execution providers in the list.\n each execution provider has the following format:\n ```json\n {\n \"Name\": \"OrtGenAI\",\n \"Config\": {\n \"device_type\": \"GPU\",\n \"device_id\": 0\n },\n \t\"Models\": [\n \t overriden models here\n ]\n }\n ```\n Where:\n - `Name`: the name of the execution provider.\n - `Config`: the configuration for the execution provider, this is specific to each execution provider, for\n example, the configuration for OrtGenAI execution provider may look like this:\n ```json\n {\n \"device_type\": \"GPU\",\n \"device_id\": 0\n }\n ```\n `Models`: this is an optional section that can be used to override specific models for the current execution\n provider. It has the same format as the `Models` section of the scenario. You need to provide the `ModelName` and\n override values for the `FilePath`, `DataFilePath`, and `TokenizerPath` if necessary.\n\n### URIs Supported\n\nThis tool accepts URIs in the following formats:\n\n- Local files: `file://\u003cpath\u003e` (must be a valid absolute local path or relative to `BaseDir` (if provided). Please note that these files are stored in cache later.)\n- Remote resources: `https://\u003curl\u003e` (must be a valid URL)\n\n### Tested Workflows\n\nYou can find specific tested workflows for different vendors in:\n\n```\ndata/configs/vendors_default/\n```\n\n## Supported Execution Providers and Configurations\n\nThe following table contains a list of the supported execution providers (EPs) and the possible configurations we\nsupport for each of them.\n\n| **Execution Provider** | **Configuration Options** | **Configuration Type** | **Required** | **Default Value** | **Available Values** | **Description**\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; | \n|-----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|--------------|-----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| NativeOpenVINO(✅ **Active**) | device_type | string | True | | [\"GPU\", \"NPU\"] | |\n| | num_of_threads | integer | False | 0 | \u003e= -1 | |\n| | num_streams | integer | False | 0 | \u003e= 1 | |\n| OrtGenAI(✅ **Active**) | device_type | string | True | | [\"GPU\"] | |\n| | device_id | integer | False | 0 | | |\n| | device_vendor | string | False | | | |\n\n### Status Definitions\n\n- ✅ **Active**: Enabled for production and development builds.\n\n### Notes\n\n- OrtGenAI:\n - `device_id` is not required. If not present, the app will use any adapters it finds, unless they are\n software/default render devices.\n - `devide_type` - optional; \"GPU\" is a valid option. If it is used:\n - If `device_id` is present, it will determine whether the device is GPU. If it does not match, it won't\n run.\n - If `device_id` is not provided, `device_type` will function as a filter, only running on devices that are \"\n GPU\" or \"NPU\".\n - `device_vendor`: It can be a vendor name, such as \"Nvidia\", \"AMD\" or \"Intel\". Devices will be filtered to match the\n vendor. Case insensitive:\n - If `device_id` is present, it will check to see if the device vendor matches. If it does not match, it will\n not run.\n - If `device_id` is not specified, device_vendor will act as a filter, only running on devices that match the\n value.\n\n## Supported Platforms by Execution Provider\n\n| Execution Provider | Windows x64 | Windows ARM | macOS | Comments |\n|--------------------|-------------|-------------|-------|----------|\n| NativeOpenVINO | ✅ | ❌ | ❌ | |\n| OrtGenAI | ✅ | ❌ | ❌ | |\n\n_____________________________________________\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fmlperf_client","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlcommons%2Fmlperf_client","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlcommons%2Fmlperf_client/lists"}