{"id":21169443,"url":"https://github.com/smarr/rebench","last_synced_at":"2025-04-12T22:19:18.559Z","repository":{"id":1318184,"uuid":"1263079","full_name":"smarr/ReBench","owner":"smarr","description":"Execute and document benchmarks reproducibly.","archived":false,"fork":false,"pushed_at":"2025-02-09T11:38:30.000Z","size":1457,"stargazers_count":90,"open_issues_count":48,"forks_count":25,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-04-12T22:19:10.568Z","etag":null,"topics":["benchmarking","continuous-benchmarking","continuous-testing","performance-tracking","python","reproducibility","research-tool","science"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smarr.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2011-01-17T10:43:28.000Z","updated_at":"2025-03-20T03:07:51.000Z","dependencies_parsed_at":"2024-03-22T23:38:17.895Z","dependency_job_id":"0eadab49-0f21-4dd7-aff3-b4b675479537","html_url":"https://github.com/smarr/ReBench","commit_stats":{"total_commits":817,"total_committers":14,"mean_commits":"58.357142857142854","dds":"0.10648714810281523","last_synced_commit":"3914b3ee5a87ea6f6c433386ba6e6f607f0dc0ae"},"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2FReBench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2FReBench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2FReBench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smarr%2FReBench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smarr","download_url":"https://codeload.github.com/smarr/ReBench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248638105,"owners_count":21137605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","continuous-benchmarking","continuous-testing","performance-tracking","python","reproducibility","research-tool","science"],"created_at":"2024-11-20T15:30:15.036Z","updated_at":"2025-04-12T22:19:18.516Z","avatar_url":"https://github.com/smarr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ReBench: Execute and Document Benchmarks Reproducibly\n\n[![Build Status](https://github.com/smarr/ReBench/workflows/CI/badge.svg)](https://github.com/smarr/ReBench/actions)\n[![PyPI version](https://badge.fury.io/py/ReBench.svg)](https://badge.fury.io/py/ReBench)\n[![Documentation](https://readthedocs.org/projects/rebench/badge/?version=latest)](https://rebench.readthedocs.io/)\n[![Downloads](https://pepy.tech/badge/rebench)](https://pepy.tech/project/rebench)\n[![Coverage](https://coveralls.io/repos/github/smarr/ReBench/badge.svg?branch=master)](https://coveralls.io/github/smarr/ReBench?branch=master)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1311762.svg)](https://doi.org/10.5281/zenodo.1311762)\n\nReBench is a tool to run and document benchmark experiments.\nCurrently, it is often used for benchmarking language implementations,\nbut it can be used to monitor the performance of all\nkinds of other applications and programs, too.\n\nThe ReBench [configuration format][docs] is a text format based on [YAML](https://yaml.org/).\nA configuration file defines how to build and execute a set of *experiments*,\ni.e. benchmarks.\nIt describes which executable was used, which parameters were given\nto the benchmarks, and the number of iterations to be used to obtain\nstatistically reliable results.\n\nWith this approach, the configuration contains all benchmark-specific\ninformation to reproduce a benchmark run. However, it does not capture\nthe whole system.\n\nThe data of all benchmark runs is recorded in a data file for later analysis.\nImportant for long-running experiments, benchmarks can be aborted and\ncontinued at a later time.\n\nReBench focuses on the execution aspect and does not provide advanced\nanalysis facilities itself. Instead, the recorded results should be processed\nby dedicated tools such as scripts for statistical analysis in R, Python, etc,\nor [ReBenchDB][1], for continuous performance tracking.\n\nThe documentation for ReBench is hosted at [https://rebench.readthedocs.io/][docs].\n\n## Goals and Features\n\nReBench is designed to\n\n - enable reproduction of experiments;\n - document all benchmark parameters;\n - provide a flexible execution model,\n   with support for interrupting and continuing benchmarking;\n - enable the definition of complex sets of comparisons and their flexible execution;\n - report results to continuous performance monitoring systems, e.g., [ReBenchDB][1];\n - provide basic support for building/compiling benchmarks/experiments on demand;\n - be extensible to parse output of custom benchmark harnesses.\n\n## ReBench Denoise\n\n[Denoise][denoise-docs] configures a Linux system for benchmarking.\nIt adapts parameters of the CPU frequency management and task scheduling\nto reduce some of the variability that can cause widely different\nbenchmark results for the same experiment.\n\nDenoise is inspired by [Krun](https://github.com/softdevteam/krun/#readme),\nwhich has many more features to carefully minimize possible interference.\nKrun is the tool of choice if the most reliable results are required.\nReBench only adapts a subset of the parameters, while staying self-contained\nand minimizing external dependencies.\n\n## Non-Goals\n\nReBench isn't\n\n - a framework for (micro)benchmarks.\n   Instead, it relies on existing harnesses and can be extended to parse their\n   output.\n - a performance analysis tool. It is meant to execute experiments and\n   record the corresponding measurements.\n - a data analysis tool. It provides only a bare minimum of statistics,\n   but has an easily parseable data format that can be processed, e.g., with R.\n\n## Installation\n\n\u003ca id=\"install\"\u003e\u003c/a\u003e\n\nReBench is implemented in Python and can be installed via pip:\n\n```bash\npip install rebench\n```\n\nTo reduce noise generated by the system, `rebench-denoise` depends on:\n\n - `sudo` rights. `rebench` will attempt to determine suitable configuration\n   parameters and suggest them. This includes allowing the execution of `rebench-denoise`\n   via `sudo` without password and with the permission to set environment variables (`SETENV`).\n - [`cpuset`](https://github.com/lpechacek/cpuset/) to reserve cores for benchmarking.\n   On Ubuntu: `apt install cpuset`\n\n`rebench-denoise` is currently tested on Ubuntu and Rocky Linux. It is designed to degrade\ngracefully and report the expected implications when it cannot adapt system\nsettings. See the [docs][denoise-docs] for details.\n\n## Usage\n\nA minimal configuration file looks like this:\n\n```yaml\n# this run definition will be chosen if no parameters are given to rebench\ndefault_experiment: all\ndefault_data_file: 'example.data'\n\n# a set of suites with different benchmarks and possibly different settings\nbenchmark_suites:\n    ExampleSuite:\n        gauge_adapter: RebenchLog\n        command: Harness %(benchmark)s %(input)s %(variable)s\n        input_sizes: [2, 10]\n        variable_values:\n            - val1\n        benchmarks:\n            - Bench1\n            - Bench2\n\n# a set of executables for the benchmark execution\nexecutors:\n    MyBin1:\n        path: bin\n        executable: test-vm1.py %(cores)s\n        cores: [1]\n    MyBin2:\n        path: bin\n        executable: test-vm2.py\n\n# combining benchmark suites and executions\nexperiments:\n    Example:\n        suites:\n          - ExampleSuite\n        executions:\n            - MyBin1\n            - MyBin2\n```\n\nSaved as `test.conf`, this configuration could be executed with ReBench as follows:\n\n```bash\nrebench test.conf\n```\n\nSee the documentation for details: [https://rebench.readthedocs.io/][docs].\n\n## Support and Contributions\n\nIn case you encounter issues,\nplease feel free to [open an issue](https://github.com/smarr/rebench/issues/new)\nso that we can help.\n\nFor contributions, we use pull requests. For larger contributions,\nit is likely useful to discuss them upfront in an issue first.\n\n### Development Setup\n\nFor the development setup, the currently recommended\nway is to use `pip install --editable .` in the root directory of the repository.\nYou may also want to use a virtual environment to avoid conflicts with other Python packages.\n\nFor instance:\n\n```bash\ngit clone https://github.com/smarr/rebench.git\ncd rebench\npip install --editable .\n```\n\nUnit tests and linting can be run with:\n\n```bash\npython -m pytest\npython -m pylint rebench\n```\n\n## Use in Academia\n\nIf you use ReBench for research and in academic publications, please consider\nciting it.\n\nThe preferred citation is:\n\n```bibtex\n@misc{ReBench:2025,\n  author = {Marr, Stefan},\n  doi = {10.5281/zenodo.1311762},\n  month = {February},\n  note = {Version 1.3},\n  publisher = {GitHub},\n  title = {ReBench: Execute and Document Benchmarks Reproducibly},\n  year = 2025\n}\n```\n\nSome publications that have been using ReBench include:\n\n - [Transient Typechecks are (Almost) Free](https://stefan-marr.de/downloads/ecoop19-roberts-et-al-transient-typechecks-are-almost-free.pdf), Roberts et al. 2019.\n - [Efficient and Deterministic Record \u0026 Replay for Actor Languages](https://stefan-marr.de/downloads/manlang18-aumayr-et-al-efficient-and-deterministic-record-and-replay-for-actor-languages.pdf), D. Aumayr et al. 2018.\n - [Fully Reflective Execution Environments: Virtual Machines for More Flexible Software](https://doi.org/10.1109%2fTSE.2018.2812715), G. Chari et al. 2018.\n - [Building efficient and highly run-time adaptable virtual machines](https://lafhis.dc.uba.ar/sites/default/files/papers/buildingEfficientReflectiveVMs.pdf), G. Chari et al. 2017.\n - [Improving live debugging of concurrent threads through thread histories](https://doi.org/10.1016/j.scico.2017.10.005), M. Leske et al. 2017.\n - [Adaptive Just-in-time Value Class Optimization for Lowering Memory Consumption and Improving Execution Time Performance](https://arxiv.org/pdf/1606.06726) T. Pape et al. 2016\n - [Cross-Language Compiler Benchmarking---Are We Fast Yet?](https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-are-we-fast-yet/), S. Marr et al. 2016.\n - [Tracing vs. Partial Evaluation: Comparing Meta-Compilation Approaches for Self-Optimizing Interpreters](https://stefan-marr.de/papers/oopsla-marr-ducasse-meta-tracing-vs-partial-evaluation/), S. Marr, S. Ducasse. 2015.\n - [Zero-Overhead Metaprogramming: Reflection and Metaobject Protocols Fast and without Compromises](https://stefan-marr.de/papers/pldi-marr-et-al-zero-overhead-metaprogramming/), S. Marr et al. 2015.\n - [Pycket: a tracing JIT for a functional language](https://www.hpi.uni-potsdam.de/hirschfeld/publications/media/BaumanBolzHirschfeldKirilichevPapeSiekTobinHochstadt_2015_PycketATracingJitForAFunctionalLanguage_AcmDL.pdf), S. Bauman et al. 2015.\n - [Adaptive just-in-time value class optimization: transparent data structure inlining for fast execution](https://hpi.uni-potsdam.de/hirschfeld/publications/media/PapeBolzHirschfeld_2015_AdaptiveJustInTimeValueClassOptimization_AcmDL.pdf), T. Pape et al. 2015.\n - [Meta-tracing makes a fast Racket](https://www.hpi.uni-potsdam.de/hirschfeld/publications/media/BolzPapeSiekTobinHochstadt_2014_MetaTracingMakesAFastRacket_AuthorsVersion.pdf), C. F. Bolz et al. 2014.\n - [Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors](https://stefan-marr.de/downloads/agere13-swalens-et-al-cloud-parte-elastic-complex-event-processing-based-on-mobile-actors.pdf), J. Swalens et al. 2013.\n - [Identifying A Unifying Mechanism for the Implementation of Concurrency Abstractions on Multi-Language Virtual Machines](https://www.stefan-marr.de/downloads/tools12-smarr-dhondt-identifying-a-unifying-mechanism-for-the-implementation-of-concurrency-abstractions-on-multi-language-virtual-machines.pdf), S. Marr, T. D'Hondt. 2012.\n - [Insertion Tree Phasers: Efficient and Scalable Barrier Synchronization for Fine-grained Parallelism](https://www.stefan-marr.de/downloads/hpcc2010-marr-etal-insertion-tree-phasers.pdf), S. Marr et al. 2011.\n\n[1]: https://github.com/smarr/ReBenchDB\n[docs]: https://rebench.readthedocs.io/\n[denoise-docs]: https://rebench.readthedocs.io/en/latest/denoise/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmarr%2Frebench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmarr%2Frebench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmarr%2Frebench/lists"}