{"id":36703789,"url":"https://github.com/sosy-lab/benchexec","last_synced_at":"2026-02-02T10:48:02.138Z","repository":{"id":27284933,"uuid":"30758422","full_name":"sosy-lab/benchexec","owner":"sosy-lab","description":"BenchExec: A Framework for Reliable Benchmarking and Resource Measurement","archived":false,"fork":false,"pushed_at":"2025-12-18T10:38:48.000Z","size":28261,"stargazers_count":274,"open_issues_count":100,"forks_count":222,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-12-19T18:37:51.749Z","etag":null,"topics":["benchmark","benchmark-framework","benchmarking","cgroups","linux","python","resource-measurement"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sosy-lab.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":".zenodo.json","notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2015-02-13T13:55:15.000Z","updated_at":"2025-12-18T10:38:51.000Z","dependencies_parsed_at":"2023-02-17T18:31:10.318Z","dependency_job_id":"eabe0830-5a3c-4f93-8f8a-e4d75f75f899","html_url":"https://github.com/sosy-lab/benchexec","commit_stats":{"total_commits":4499,"total_committers":184,"mean_commits":24.45108695652174,"dds":0.5121138030673482,"last_synced_commit":"41b77d311da80a44b3b9ad32dc3eaf49501448c6"},"previous_names":[],"tags_count":70,"template":false,"template_full_name":null,"purl":"pkg:github/sosy-lab/benchexec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sosy-lab%2Fbenchexec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sosy-lab%2Fbenchexec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sosy-lab%2Fbenchexec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sosy-lab%2Fbenchexec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sosy-lab","download_url":"https://codeload.github.com/sosy-lab/benchexec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sosy-lab%2Fbenchexec/sbom","scorecard":{"id":838791,"data":{"date":"2025-08-11","repo":{"name":"github.com/sosy-lab/benchexec","commit":"09524cb354f7d39876b676fc61964f80bbb3b9c8"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.4,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Maintained","score":10,"reason":"30 commit(s) and 7 issue activity found in the last 90 days -- score normalized to 10","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Code-Review","score":2,"reason":"Found 5/22 approved changesets -- score normalized to 2","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/black.yml:1","Warn: no topLevel permission defined: .github/workflows/javascript.yml:1","Warn: no topLevel permission defined: .github/workflows/reuse.yml:1","Warn: no topLevel permission defined: .github/workflows/ruff.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":8,"reason":"5 out of the last 5 releases have a total of 5 signed artifacts.","details":["Info: signed release artifact: benchexec-3.30-py3-none-any.whl.asc: https://github.com/sosy-lab/benchexec/releases/tag/3.30","Info: signed release artifact: benchexec-3.29-py3-none-any.whl.asc: https://github.com/sosy-lab/benchexec/releases/tag/3.29","Info: signed release artifact: BenchExec-3.28-py3-none-any.whl.asc: https://github.com/sosy-lab/benchexec/releases/tag/3.28","Info: signed release artifact: BenchExec-3.27-py3-none-any.whl.asc: https://github.com/sosy-lab/benchexec/releases/tag/3.27","Info: signed release artifact: BenchExec-3.26-py3-none-any.whl.asc: https://github.com/sosy-lab/benchexec/releases/tag/3.26","Warn: release artifact 3.30 does not have provenance: https://api.github.com/repos/sosy-lab/benchexec/releases/223575971","Warn: release artifact 3.29 does not have provenance: https://api.github.com/repos/sosy-lab/benchexec/releases/205062572","Warn: release artifact 3.28 does not have provenance: https://api.github.com/repos/sosy-lab/benchexec/releases/203159251","Warn: release artifact 3.27 does not have provenance: https://api.github.com/repos/sosy-lab/benchexec/releases/187104330","Warn: release artifact 3.26 does not have provenance: https://api.github.com/repos/sosy-lab/benchexec/releases/183534472"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 13 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Pinned-Dependencies","score":1,"reason":"dependency not pinned by hash detected -- score normalized to 1","details":["Info: Possibly incomplete results: error parsing shell code: \u003e must be followed by a word: debian/additional_files/usr/share/benchexec/apparmor.d/benchexec:0","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/black.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/black.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/black.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/black.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:28: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:47: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:52: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:57: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:76: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:81: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/javascript.yml:86: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/javascript.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/reuse.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/reuse.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/reuse.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/reuse.yml/main?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/ruff.yml:16: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/ruff.yml/main?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/ruff.yml:19: update your workflow using https://app.stepsecurity.io/secureworkflow/sosy-lab/benchexec/ruff.yml/main?enable=pin","Warn: containerImage not pinned by hash: contrib/p4/docker_files/basic_node/Dockerfile:11: pin your Docker image by updating ubuntu:18.04 to ubuntu:18.04@sha256:152dc042452c496007f07ca9127571cb9c29697f42acbfad72324b2bb2e43c98","Warn: containerImage not pinned by hash: contrib/p4/docker_files/ptf_tester/Dockerfile:10: pin your Docker image by updating ubuntu:18.04 to ubuntu:18.04@sha256:152dc042452c496007f07ca9127571cb9c29697f42acbfad72324b2bb2e43c98","Warn: containerImage not pinned by hash: contrib/p4/docker_files/switch_bmv2/Dockerfile:10: pin your Docker image by updating ubuntu:16.04 to ubuntu:16.04@sha256:1f1a2d56de1d604801a9671f301190704c25d604a416f59e03c04f5c6ffee0d6","Warn: containerImage not pinned by hash: test/Dockerfile.python-3.10:16: pin your Docker image by updating python:3.10 to python:3.10@sha256:c69e40390fbbcbe98785d0bec0e93a9308850c6727edba2293bba4c79bdaffb5","Warn: containerImage not pinned by hash: test/Dockerfile.python-3.11:16: pin your Docker image by updating python:3.11 to python:3.11@sha256:0dab67c838514eef83e6c9d2c0e53e960fc94237635e8996d08caeec98937abc","Warn: containerImage not pinned by hash: test/Dockerfile.python-3.12:16: pin your Docker image by updating python:3.12 to python:3.12@sha256:23e642ca5600c3c6925490bf500914abe411d944b237ac0c6c87af2cd091b8fd","Warn: containerImage not pinned by hash: test/Dockerfile.python-3.13:16: pin your Docker image by updating python:3.13 to python:3.13@sha256:3b2f1b9c9948e9dc96e1a2f4668ba9870ff43ab834f91155697476142b3bc299","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/basic_node/Dockerfile:24","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/basic_node/Dockerfile:25","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/basic_node/Dockerfile:26","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/basic_node/Dockerfile:28","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/ptf_tester/Dockerfile:17","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/ptf_tester/Dockerfile:21","Warn: pipCommand not pinned by hash: contrib/p4/docker_files/ptf_tester/Dockerfile:26","Warn: pipCommand not pinned by hash: test/Dockerfile.python-3.10:28-35","Warn: pipCommand not pinned by hash: test/Dockerfile.python-3.11:28-35","Warn: pipCommand not pinned by hash: test/Dockerfile.python-3.12:28-35","Warn: pipCommand not pinned by hash: test/Dockerfile.python-3.13:28-36","Warn: pipCommand not pinned by hash: release.sh:83","Warn: pipCommand not pinned by hash: release.sh:84","Info:   0 out of  12 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   3 third-party GitHubAction dependencies pinned","Info:   0 out of   7 containerImage dependencies pinned","Info:   0 out of  13 pipCommand dependencies pinned","Info:   3 out of   3 npmCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Vulnerabilities","score":0,"reason":"43 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-968p-4wvh-cqc8","Warn: Project is vulnerable to: GHSA-67hx-6x53-jw92","Warn: Project is vulnerable to: GHSA-qwcr-r2fm-qrc7","Warn: Project is vulnerable to: GHSA-v6h2-p8h4-qcjw","Warn: Project is vulnerable to: GHSA-grv7-fg5c-xmjg","Warn: Project is vulnerable to: GHSA-pxg6-pf52-xh8x","Warn: Project is vulnerable to: GHSA-3xgq-45jj-v275","Warn: Project is vulnerable to: GHSA-36jr-mh4h-2g58","Warn: Project is vulnerable to: GHSA-ghr5-ch3p-vcr6","Warn: Project is vulnerable to: GHSA-rv95-896h-c2vc","Warn: Project is vulnerable to: GHSA-qw6h-vgh9-j6wx","Warn: Project is vulnerable to: GHSA-jchw-25xp-jwwc","Warn: Project is vulnerable to: GHSA-cxjh-pqwp-8mfp","Warn: Project is vulnerable to: GHSA-fjxv-7rqg-78g4","Warn: Project is vulnerable to: GHSA-c429-5p7v-vgjp","Warn: Project is vulnerable to: GHSA-c7qv-q95q-8v27","Warn: Project is vulnerable to: GHSA-4www-5p9h-95mh","Warn: Project is vulnerable to: GHSA-9gqv-wp59-fq42","Warn: Project is vulnerable to: GHSA-9c47-m6qq-7p4h","Warn: Project is vulnerable to: GHSA-76p3-8jx3-jpfq","Warn: Project is vulnerable to: GHSA-3rfm-jhwj-7488","Warn: Project is vulnerable to: GHSA-hhq3-ff78-jv3g","Warn: Project is vulnerable to: GHSA-952p-6rrq-rcjv","Warn: Project is vulnerable to: GHSA-f8q6-p94x-37v3","Warn: Project is vulnerable to: GHSA-mwcw-c2x4-8c55","Warn: Project is vulnerable to: GHSA-rp65-9cf3-cjxr","Warn: Project is vulnerable to: GHSA-76c9-3jph-rj3q","Warn: Project is vulnerable to: GHSA-9wv6-86v2-598j","Warn: Project is vulnerable to: GHSA-rhx6-c78j-4q9w","Warn: Project is vulnerable to: GHSA-7fh5-64p2-3v2j","Warn: Project is vulnerable to: GHSA-gcx4-mw62-g8wm","Warn: Project is vulnerable to: GHSA-c2qf-rxjj-qqgw","Warn: Project is vulnerable to: GHSA-m6fv-jmcg-4jfg","Warn: Project is vulnerable to: GHSA-76p7-773f-r4q5","Warn: Project is vulnerable to: GHSA-cm22-4g7w-348p","Warn: Project is vulnerable to: GHSA-72xf-g2v4-qvf3","Warn: Project is vulnerable to: GHSA-hc6q-2mpp-qw7j","Warn: Project is vulnerable to: GHSA-4vvj-4cpr-p986","Warn: Project is vulnerable to: GHSA-wr3j-pwj9-hqq6","Warn: Project is vulnerable to: GHSA-4v9v-hfq4-rm2v","Warn: Project is vulnerable to: GHSA-9jgg-88mc-972h","Warn: Project is vulnerable to: GHSA-j8xg-fqg3-53r7","Warn: Project is vulnerable to: GHSA-3h5v-q93c-6h6q"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-23T19:54:00.902Z","repository_id":27284933,"created_at":"2025-08-23T19:54:00.903Z","updated_at":"2025-08-23T19:54:00.903Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28338972,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T10:58:46.209Z","status":"ssl_error","status_checked_at":"2026-01-12T10:58:42.742Z","response_time":98,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmark-framework","benchmarking","cgroups","linux","python","resource-measurement"],"created_at":"2026-01-12T11:40:44.530Z","updated_at":"2026-01-12T11:40:44.594Z","avatar_url":"https://github.com/sosy-lab.png","language":"Python","funding_links":[],"categories":["Software Artifact \u0026 Technical Guides"],"sub_categories":[],"readme":"\u003c!--\nThis file is part of BenchExec, a framework for reliable benchmarking:\nhttps://github.com/sosy-lab/benchexec\n\nSPDX-FileCopyrightText: 2007-2020 Dirk Beyer \u003chttps://www.sosy-lab.org\u003e\n\nSPDX-License-Identifier: Apache-2.0\n--\u003e\n\n# BenchExec\n\n## A Framework for Reliable Benchmarking and Resource Measurement\n\n[![Build Status](https://gitlab.com/sosy-lab/software/benchexec/badges/main/pipeline.svg)](https://gitlab.com/sosy-lab/software/benchexec/pipelines)\n[![Apache 2.0 License](https://img.shields.io/badge/license-Apache--2-brightgreen.svg)](https://www.apache.org/licenses/LICENSE-2.0)\n[![PyPI version](https://img.shields.io/pypi/v/BenchExec.svg)](https://pypi.python.org/pypi/BenchExec)\n[![DOI](https://zenodo.org/badge/30758422.svg)](https://zenodo.org/badge/latestdoi/30758422)\n\n**News and Updates**:\n- BenchExec 3.25 can use [fuse-overlayfs](https://github.com/containers/fuse-overlayfs/) to make container mode work out-of-the-box again!\n- BenchExec 3.18 brings support for systems with cgroups v2!\n- We now provide an [Ubuntu PPA](https://launchpad.net/~sosy-lab/+archive/ubuntu/benchmarking) that makes installing and upgrading BenchExec easier ([docs](https://github.com/sosy-lab/benchexec/blob/main/doc/INSTALL.md#debianubuntu)).\n- An extended version of our paper on BenchExec and its background was published as open access in the journal STTT,\n  you can read [Reliable Benchmarking: Requirements and Solutions](https://doi.org/10.1007/s10009-017-0469-y) online.\n  We also provide a set of [overview slides](https://www.sosy-lab.org/research/prs/Latest_ReliableBenchmarking.pdf).\n\n\u003e To help new or inexperienced users get started with reliable benchmarking\n\u003e right away, we offer a [quickstart guide](doc/quickstart.md) that contains\n\u003e a brief explanation of the issues of common setups as well as the (few)\n\u003e steps necessary to setup and use BenchExec instead.\n\nBenchExec is a framework for reliable benchmarking and resource measurement\nand provides a standalone solution for benchmarking\nthat takes care of important low-level details for accurate, precise, and reproducible measurements\nas well as result handling and analysis for large sets of benchmark runs.\nHowever, even users of other benchmarking frameworks or scripts\ncan benefit from BenchExec\nby letting it perform the resource measurements and limits\ninstead of less reliable tools and techniques like `time` or `ulimit`,\nand results produced by BenchExec can easily be exported for use with other tools.\n\nIn particular, BenchExec provides three major features:\n\n- execution of arbitrary commands with precise and reliable measurement\n  and limitation of resource usage (e.g., CPU time and memory),\n  and isolation against other running processes  \n  (provided by [`runexec`](https://github.com/sosy-lab/benchexec/blob/main/doc/runexec.md),\n  a replacement for `time` and similar tools)\n- an easy way to define benchmarks with specific tool configurations\n  and resource limits,\n  and automatically executing them on large sets of input files  \n  (provided by [`benchexec`](https://github.com/sosy-lab/benchexec/blob/main/doc/benchexec.md)\n  on top of `runexec`)\n- generation of interactive tables and plots for the results  \n  (provided by [`table-generator`](https://github.com/sosy-lab/benchexec/blob/main/doc/table-generator.md)\n  for results produced with `benchexec`)\n\n\nUnlike other benchmarking frameworks,\nBenchExec is able to reliably measure and limit resource usage\nof the benchmarked tool even if the latter spawns subprocesses.\nIn order to achieve this,\nit uses the [cgroups feature](https://docs.kernel.org/admin-guide/cgroup-v2.html)\nof the Linux kernel to correctly handle groups of processes.\nFor proper isolation of the benchmarks, it uses (if available)\nLinux [user namespaces](http://man7.org/linux/man-pages/man7/namespaces.7.html)\nand an overlay filesystem\n(either [kernel-based](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt)\nor [fuse-overlayfs](https://github.com/containers/fuse-overlayfs/))\nto create a [container](https://github.com/sosy-lab/benchexec/blob/main/doc/container.md)\nthat restricts interference of the executed tool with the benchmarking host.\nMore information on why this is necessary and the problems with other tools\ncan be found in our paper\n[Reliable Benchmarking: Requirements and Solutions](https://doi.org/10.1007/s10009-017-0469-y) (open access)\nand our [slides](https://www.sosy-lab.org/research/prs/Latest_ReliableBenchmarking.pdf)\n(starting with slide \"Checklist\").\n\nBenchExec is intended for benchmarking non-interactive tools on Linux systems.\nIt measures CPU time, wall time, and memory usage of a tool,\nand allows to specify limits for these resources.\nIt also allows to limit the CPU cores and (on NUMA systems) memory regions,\nand the container mode allows to restrict filesystem and network access.\nIn addition to measuring resource usage,\nBenchExec can optionally verify that the result of the tool was as expected\nand extract further statistical data from the output.\nResults from multiple runs can be combined into CSV and interactive HTML tables,\nof which the latter provide scatter and quantile plots\n(have a look at our [demo table](https://sosy-lab.github.io/benchexec/example-table/svcomp-simple-cbmc-cpachecker.table.html)).\n\nBenchExec works only on Linux and needs a one-time setup of cgroups by the machine's administrator.\nThe actual benchmarking can be done by any user and does not need root access.\n\nBenchExec was originally developed for use with the software verification framework\n[CPAchecker](https://cpachecker.sosy-lab.org)\nand is now developed as an independent project\nat the [Software Systems Lab](https://www.sosy-lab.org)\nof the [Ludwig-Maximilians-Universität München (LMU Munich)](https://www.uni-muenchen.de).\n\n### Links\n\n- [Documentation](https://github.com/sosy-lab/benchexec/tree/main/doc/INDEX.md)\n- [Demo](https://sosy-lab.github.io/benchexec/example-table/svcomp-simple-cbmc-cpachecker.table.html) of a result table\n- [Downloads](https://github.com/sosy-lab/benchexec/releases)\n- [Changelog](https://github.com/sosy-lab/benchexec/tree/main/CHANGELOG.md)\n- [BenchExec GitHub Repository](https://github.com/sosy-lab/benchexec),\n  use this for [reporting issues and asking questions](https://github.com/sosy-lab/benchexec/issues)\n- [BenchExec at PyPI](https://pypi.python.org/pypi/BenchExec)\n- [BenchExec at Zenodo](https://doi.org/10.5281/zenodo.1163552): Each release gets a DOI such that you can reference it from your publications and artifacts.\n\n### Literature\n\n- [Reliable Benchmarking: Requirements and Solutions](https://doi.org/10.1007/s10009-017-0469-y), by D. Beyer, S. Löwe, and P. Wendler.  International Journal on Software Tools for Technology Transfer (STTT), 21(1):1-29, 2019. [doi:10.1007/s10009-017-0469-y](https://doi.org/10.1007/s10009-017-0469-y). Journal paper about BenchExec (open access, also with a [supplementary webpage](https://www.sosy-lab.org/research/benchmarking/) and [overview slides](https://www.sosy-lab.org/research/prs/Latest_ReliableBenchmarking.pdf))\n- [CPU Energy Meter: A Tool for Energy-Aware Algorithms Engineering](https://doi.org/10.1007/978-3-030-45237-7_8), by D. Beyer and P. Wendler. In Proc. TACAS 2020, part 2, LNCS 12079, pages 126-133, 2020. Springer. [doi:10.1007/978-3-030-45237-7_8](https://doi.org/10.1007/978-3-030-45237-7_8)\n- [Benchmarking and Resource Measurement](https://doi.org/10.1007/978-3-319-23404-5_12), by D. Beyer, S. Löwe, and P. Wendler. In Proc. SPIN 2015, LNCS 9232, pages 160-178, 2015. Springer. [doi:10.1007/978-3-319-23404-5_12](https://doi.org/10.1007/978-3-319-23404-5_12)\n\n### License and Copyright\n\nBenchExec is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0),\ncopyright [Dirk Beyer](https://www.sosy-lab.org/people/beyer/).\nExceptions are some tool-info modules\nand third-party code that is bundled in the HTML tables,\nwhich are available under several other free licenses\n(cf. [folder `LICENSES`](https://github.com/sosy-lab/benchexec/tree/main/LICENSES)).\n\n### Authors\nMaintainer: [Philipp Wendler](https://www.philippwendler.de)\n\nContributors:\n- [Eshaan Aggarwal](https://github.com/EshaanAgg)\n- [Aditya Arora](https://github.com/alohamora)\n- [Levente Bajczi](https://github.com/leventeBajczi)\n- [Dirk Beyer](https://www.sosy-lab.org/people/beyer/)\n- [Laura Bschor](https://github.com/laurabschor)\n- [Thomas Bunk](https://github.com/TBunk)\n- [Montgomery Carter](https://github.com/MontyCarter)\n- [Po-Chun Chien](https://github.com/Po-Chun-Chien)\n- [Andreas Donig](https://github.com/adonig)\n- [Florian Eder](https://github.com/schroeding)\n- [Karlheinz Friedberger](https://www.sosy-lab.org/people/friedberger)\n- [Robin Gloster](https://github.com/globin)\n- [Sam Grayson](https://github.com/charmoniumQ)\n- Peter Häring\n- [Florian Heck](https://github.com/fheck)\n- [Chinmay Joshi](https://github.com/JawHawk)\n- [George Karpenkov](http://metaworld.me/)\n- [Mike Kazantsev](http://fraggod.net/)\n- [Hugo van Kemenade](https://github.com/hugovk)\n- [Tobias Kleinert](https://github.com/Sowasvonbot)\n- [Michael Lachner](https://github.com/lachnerm)\n- [Thomas Lemberger](https://www.sosy-lab.org/people/lemberger/)\n- [Lorenz Leutgeb](https://github.com/lorenzleutgeb)\n- [Sebastian Ott](https://github.com/ottseb)\n- Stefan Löwe\n- [Stephan Lukasczyk](https://github.com/stephanlukasczyk)\n- [Tobias Meggendorfer](https://github.com/incaseoftrouble)\n- Alexander von Rhein\n- [Simmo Saan](https://github.com/sim642)\n- [Alexander Schremmer](https://www.xing.com/profile/Alexander_Schremmer)\n- [Dennis Simon](https://github.com/DennisSimon)\n- [Andreas Stahlbauer](http://stahlbauer.net/)\n- [Thomas Stieglmaier](https://stieglmaier.me/)\n- [Martin Yankov](https://github.com/marto97)\n- [Hojan Young](https://github.com/younghojan)\n- [Ilja Zakharov](https://github.com/IljaZakharov)\n- and [lots of more people who integrated tools into BenchExec](https://github.com/sosy-lab/benchexec/graphs/contributors)\n\n### Users of BenchExec\n\nSeveral well-known international competitions use BenchExec,\nsuch as [SMT-COMP](https://smt-comp.github.io/),\n[SV-COMP](https://sv-comp.sosy-lab.org) (software verification),\nthe [Termination Competition](https://termination-portal.org/wiki/Termination_Competition),\nand\n[Test-Comp](https://test-comp.sosy-lab.org).\nIn particular in SV-COMP\nBenchExec was used successfully for benchmarking in all instances of the competition\nand with a wide variety of benchmarked tools and millions of benchmark runs per year.\nBenchExec is also integrated into the cluster-based logic-solving service\n[StarExec](https://www.starexec.org/starexec/public/about.jsp) ([GitHub](https://github.com/StarExec/StarExec)).\n\nThe developers of the following tools use BenchExec:\n\n- [CPAchecker](https://cpachecker.sosy-lab.org), also for regression testing\n- [Dartagnan](https://github.com/hernanponcedeleon/Dat3M)\n- [ESBMC](https://github.com/esbmc/esbmc), also for regression testing and even with a [GitHub action](https://github.com/esbmc/esbmc/blob/master/.github/workflows/benchexec.yml) for BenchExec\n- [SMACK](https://github.com/smackers/smack)\n- [SMTInterpol](https://github.com/ultimate-pa/smtinterpol)\n- [TriCera](https://github.com/uuverifiers/tricera)\n- [Ultimate](https://github.com/ultimate-pa/ultimate)\n\nIf you would like to be listed here, [contact us](https://github.com/sosy-lab/benchexec/issues/new).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsosy-lab%2Fbenchexec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsosy-lab%2Fbenchexec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsosy-lab%2Fbenchexec/lists"}