{"id":26148973,"url":"https://github.com/originq/benchpress","last_synced_at":"2026-02-09T03:33:04.349Z","repository":{"id":271848725,"uuid":"914724116","full_name":"OriginQ/BenchPress","owner":"OriginQ","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-01T02:38:14.000Z","size":37562,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-08-13T03:37:23.043Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"OpenQASM","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OriginQ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-10T07:13:28.000Z","updated_at":"2025-05-04T23:01:28.000Z","dependencies_parsed_at":"2025-08-13T03:37:28.539Z","dependency_job_id":null,"html_url":"https://github.com/OriginQ/BenchPress","commit_stats":null,"previous_names":["originq/benchpress"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OriginQ/BenchPress","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OriginQ%2FBenchPress","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OriginQ%2FBenchPress/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OriginQ%2FBenchPress/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OriginQ%2FBenchPress/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OriginQ","download_url":"https://codeload.github.com/OriginQ/BenchPress/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OriginQ%2FBenchPress/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29255602,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-09T03:07:45.136Z","status":"ssl_error","status_checked_at":"2026-02-09T03:07:24.123Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-11T05:22:24.135Z","updated_at":"2026-02-09T03:33:04.322Z","avatar_url":"https://github.com/OriginQ.png","language":"OpenQASM","funding_links":[],"categories":[],"sub_categories":[],"readme":"![Benchpress Image - © 2024 IBM. All Rights Reserved.](https://github.com/user-attachments/assets/67b82edb-52d6-47ac-a513-1b7129c20ea5)\n\nQuantum software benchmarking\n___\n\n## What is Benchpress?\n\nBenchpress is an open-source tool for benchmarking quantum software.\n\nThe Benchpress open-source benchmarking suite comprises over 1,000 different tests. These are standardized benchmarking tests designed by other members of the quantum community. For example, Benchpress compares SDKs’ abilities to generate [QASMBench circuits](https://github.com/pnnl/QASMBench), [Feynman circuits](https://github.com/meamy/feynman), and [Hamiltonian circuits](https://arxiv.org/pdf/2306.13126) (https://portal.nersc.gov/cfs/m888/dcamps/hamlib/). It also includes tests designed to test a language's ability to transpiler circuits for specific hardware, including the heavy hex architecture of IBM quantum processors and other generic qubit layouts.\n\nIf you find an issue with the testing or how we completed it, we encourage you to make a pull request.\n\n## Supported SDKs\n\nBenchpress currently supports the following SDKs:\n\n- **BQSKit** (https://github.com/BQSKit/bqskit)\n- **Braket** (https://github.com/amazon-braket/amazon-braket-sdk-python)\n- **Cirq** (https://github.com/quantumlib/Cirq)\n- **Qiskit** (https://github.com/Qiskit/qiskit)\n- **Qiskit IBM transpiler** (https://github.com/Qiskit/qiskit-ibm-transpiler)\n- **Staq** (https://github.com/softwareQinc/staq)\n- **Tket** (https://github.com/CQCL/tket)\n\n## Testing resource requirements\n\nRunning Benchpress is resource intensive.  Although the exact requirements depend on the SDK in question, a full execution of all the SDKs requires a system with 96+Gb of memory and, in some cases, will consume as many CPU resources as are available / assigned.  In addition, each suite of tests takes a non-negligible about of time, typically several hours or more depending on the machine and timeout specified.\n\n## Installation\n\nBenchpress itself requires no installation.  However running it requires the tools in `requirements.txt`.  In addition, running each of the frameworks has its own dependencies in the corresponding `*-requirements.txt` file\n\n### [pre-running] Create a skiplist\n\nWith the parameter `--timeout-skip-list=\u003cSECs\u003e`, a  *skiplist* (a list of tests to skip, given they take too long) is created.\nFor example, the following line runs the tests in `benchpress/tket_gym/construct` with a 1 hour timeout:\n\n```bash\npython -m pytest  --timeout-skip-list=3600 benchpress/tket_gym/construct\n```\n\nThis will create a `skipfile.txt` file.\nThe mere existence of this file skips the tests listed there in the following executions.\nNo modifier needed.\n\n## Running the benchmark tests\n\nTo run the benchmarks in the default configuration from inside the environment in which you want to perform the tests run:\n\n```bash\npython -m pytest benchpress/*_gym\n```\nwhere `*` is one of the frameworks that you want to test, and which matches the environment you are in.\n\nTo run the benchmarks and save to JSON one can do:\n\n```bash\npython -m pytest --benchmark-save=SAVED_NAME  benchpress/*_gym\n```\nwhich will save the file to the CWD in the `.benchmarks` folder\n\nFurther details on using `pytest-benchmark` can be found here: https://pytest-benchmark.readthedocs.io/en/latest/usage.html\n\n\n## :construction: Running the memory tests :construction:\n\nBenchmarking the amount of memory a test uses can be very costly in terms of time and memory.  Here we use the `pytest-memray` plugin.  Calling the memory bechmark looks like:\n\n```bash\n\npython -m pytest --memray --trace-python-allocators --native --most-allocations=100 --benchmark-disable benchpress/*_gym\n```\n\nHere `--memray` turns on the memory profiler, `--trace-python-allocators` tracks all the memoryu allocations from Python, `--native` track C/C++/Rust memory, `--most-allocations=N` shows only the top `N` tests in terms of memory consuption, and finally `--benchmark-disable` turns off the timing benchmarks.\n\n### Histogram issues\n\nThe `pytest-memray` plugin will sometimes raise on building the histrogram included in the report by default.  Currently the only way around this error, which does not affect the tests, is to manually comment out L322 and L323 from the `plugin.py` file:\n\n```python\n#histogram_txt = cli_hist(sizes, bins=min(len(sizes), N_HISTOGRAM_BINS))\n#writeln(f\"\\t 📊 Histogram of allocation sizes: |{histogram_txt}|\")\n```\n## Testing details\n\nWe have designed Benchpress in a manner to allow all tests to be executed on each SDK, regardless of whether that functionality is supported or not.  This is facilitated by the use of \"workouts\" that define abstract base classes that define each set of tests.  This design choice has the advantage of explicitly measuring the breadth of SDK functionality\n\n### Test status description\n\nIn Benchpress each test status has a well defined meaning:\n\n- **PASSED** - Indicates that the SDK has the functionality required to run the test, and doing so completed without error, and within the desired time-limit.\n\n- **SKIPPED** - The SDK does not have the required functionality to execute the test.  This is the default for all tests defined in the workouts.\n\n- **FAILED** - The SDK has the necessary functionality, but the test failed or the test did not complete within the set time-limit.\n\n- **XFAIL** - The test fails in an irrecoverable manner, and is therefore tagged as failed rather than being executed. E.g. the test tries to use more memory than is available.\n\n### Test runtime\n\nRunning the full suite of tests will easily take a week or more if executed in serial, e.g. so that memory bandwidth or multiprocessing usage does no skew results.  Users can always select a subset of tests to reduce this overall time.\n\n## Open-source packages\n\nBenchpress makes use of files from the following open-source packages under terms of their licenses. License files are included in the corresponding directories.\n\n- [Feynman](https://github.com/meamy/feynman)\n\n- [QasmBench](https://github.com/pnnl/QASMBench)\n\n- [HamLib](https://portal.nersc.gov/cfs/m888/dcamps/hamlib/)\n\n\n## License\n\n[Apache License 2.0](LICENSE.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foriginq%2Fbenchpress","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foriginq%2Fbenchpress","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foriginq%2Fbenchpress/lists"}