{"id":18521769,"url":"https://github.com/simphotonics/benchmark_runner","last_synced_at":"2025-04-09T09:33:24.847Z","repository":{"id":100483070,"uuid":"300736518","full_name":"simphotonics/benchmark_runner","owner":"simphotonics","description":"A library for writing inline micro-benchmarks, reporting score statistics, and running sync/async benchmarks.","archived":false,"fork":false,"pushed_at":"2024-09-17T15:07:07.000Z","size":3114,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T04:43:24.932Z","etag":null,"topics":["async","benchmark","dart","execution-time","runner","score","statistics"],"latest_commit_sha":null,"homepage":"https://pub.dev/packages/benchmark_runner","language":"Dart","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simphotonics.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-02T21:09:03.000Z","updated_at":"2024-07-03T18:48:04.000Z","dependencies_parsed_at":"2024-04-17T17:55:25.615Z","dependency_job_id":"5147871d-437f-4c70-9dda-f01366b8d21d","html_url":"https://github.com/simphotonics/benchmark_runner","commit_stats":null,"previous_names":["simphotonics/benchmark_runner"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simphotonics%2Fbenchmark_runner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simphotonics%2Fbenchmark_runner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simphotonics%2Fbenchmark_runner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simphotonics%2Fbenchmark_runner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simphotonics","download_url":"https://codeload.github.com/simphotonics/benchmark_runner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248012864,"owners_count":21033254,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","benchmark","dart","execution-time","runner","score","statistics"],"created_at":"2024-11-06T17:27:33.037Z","updated_at":"2025-04-09T09:33:23.951Z","avatar_url":"https://github.com/simphotonics.png","language":"Dart","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmark Runner\n[![Dart](https://github.com/simphotonics/benchmark_runner/actions/workflows/dart.yml/badge.svg)](https://github.com/simphotonics/benchmark_runner/actions/workflows/dart.yml)\n\n\n## Introduction\n\nBenchmarking is used to estimate and compare the execution speed of\nnumerical algorithms and programs.\nThe package [`benchmark_runner`][benchmark_runner] is based on\n[`benchmark_harness`][benchmark_harness] and includes helper\nfunctions for writing *inline* micro-benchmarks with the option of\nprinting a score **histogram** and reporting the score **mean** \u0026#177;\n**standard deviation**, and score **median** \u0026#177; **inter quartile range**.\n\nThe benchmark runner allows executing several benchmark files and reports if\nuncaught exceptions/errors were encountered.\n\n## Usage\n\nInclude [`benchmark_runner`][benchmark_runner] as a `dev_dependency`\n in your `pubspec.yaml` file.\n\nWrite inline benchmarks using the functions:\n * [`benchmark`][benchmark]: Creates and runs a synchronous benchmark and\n   reports the benchmark score.\n * [`asyncBenchmark`][asyncBenchmark]: Creates and runs an\n   asynchronous benchmark.\n * [`group`][group]: Used to label a group of benchmarks.\n   The callback `body` usually contains one or several calls to\n   [`benchmark`][benchmark] and [`asyncBenchmark`][asyncBenchmark].\n   Benchmark groups may not be nested.\n * Benchmark files must end with `_benchmark.dart` in order to be detected\n   by the `benchmark_runner`.\n\n The example below shows a benchmark file containing synchronous and\n asynchronous benchmarks.\n\n  ```Dart\n  // ignore_for_file: unused_local_variable\n\n  import 'package:benchmark_runner/benchmark_runner.dart';\n\n  /// Returns the value [t] after waiting for [duration].\n  Future\u003cT\u003e later\u003cT\u003e(T t, [Duration duration = Duration.zero]) {\n    return Future.delayed(duration, () =\u003e t);\n  }\n\n  void main(List\u003cString\u003e args) async {\n    await group('Wait for duration', () async {\n      await asyncBenchmark('10ms', () async {\n        await later\u003cint\u003e(39, Duration(milliseconds: 10));\n      });\n\n      await asyncBenchmark('5ms', () async {\n        await later\u003cint\u003e(27, Duration(milliseconds: 5));\n      }, emitStats: false);\n    });\n\n    group('Set', () async {\n      await asyncBenchmark('error test', () {\n        throw ('Thrown in benchmark.');\n      });\n\n      benchmark('construct', () {\n        final set = {for (var i = 0; i \u003c 1000; ++i) i};\n      });\n\n      throw 'Error in group';\n    });\n  }\n\n  ```\n### 1. Running a Single Benchmark File\nA *single* benchmark file may be run as a Dart executable:\n```Console\n$ dart benchmark/example_async_benchmark.dart\n```\n![Console Output Single](https://raw.githubusercontent.com/simphotonics/benchmark_runner/main/images/console_output_single.png)\n\nThe console output is shown above. The following colours and coding\nare used:\n* The first column shows the micro-benchmark runtime, followed by the group\n  name and the benchmark name.\n* The labels of asynchronous groups and benchmarks are marked with an hour-glass\nsymbol.\n* The *mean* and the histogram block containing the *mean*\nare printed using \u003cspan style=\"color:#11A874\"\u003e*green*\u003c/span\u003e foreground.\n* The *median* and the block containg the *median* are printed\nusing \u003cspan style=\"color:#2370C4\"\u003e*blue*\u003c/span\u003e foreground.\n* If the same block contains mean and median then it is printed\nusing \u003cspan style=\"color:#28B5D7\"\u003e*cyan*\u003c/span\u003e foreground.\n* Errors are printed using \u003cspan style=\"color:#CB605E\"\u003e *red* \u003c/span\u003e foreground.\n\n### 2. Running Several Benchmark Files\nTo run *several* benchmark files (with the format`*_benchmark.dart`)\ninvoke the benchmark_runner and specify a directory.\nIf no directory is specified, it defaults to `benchmark`:\n\n```Console\n$ dart run benchmark_runner\n```\n\n![Console Output](https://raw.githubusercontent.com/simphotonics/benchmark_runner/main/images/console_output.png)\n\nA typical console output is shown above. In this example, the benchmark_runner\ndetected two benchmark files, ran the micro-benchmarks and produced a report.\n\n\n* The summary shows the total number of completed benchmarks, the number of\nbenchmarks with errors and the number of groups with errors (that do not\noccur within the scope of a benchmark function).\n* To show a stack trace for each error, run the benchmark_runner using\nthe option ``-v`` or `--verbose`.\n* The total benchmark run time may be shorter than the sum of the\nmicro-benchmark run times since each executable benchmark file is run in\na separate process.\n\n\n## Tips and Tricks\n\n- The scores reported by [`benchmark`][benchmark] and\n[`asyncBenchmark`][asyncBenchmark]\nrefer to a *single* run of the benchmarked function.\n\n- Benchmarks do *not* need to be enclosed by a group.\n\n- A benchmark group may *not* contain another benchmark group.\n\n- The program does **not** check for group *description*\nand benchmark *description* clashes. It can be useful to have a second\nbenchmark with the same name for example to compare the standard score\nas reported by [`benchmark_harness`][benchmark_harness] and the\nscore statistics.\n\n- By default, [`benchmark`][benchmark] and\n[`asyncBenchmark`][asyncBenchmark] report score statistics. In order to generate\nthe report provided by [`benchmark_harness`][benchmark_harness] use the\noptional argument `emitStats: false`.\n\n- Color output can be switched off by using the option: `--isMonochrome` when\ncalling the benchmark runner. When executing a single benchmark file the\ncorresponding option is `--define=isMonochrome=true`.\n\n- The default colors used to style benchmark reports are best suited\nfor a dark terminal background.\nThey can, however, be altered by setting the static variables defined by\nthe class [`ColorProfile`][ColorProfile]. In the example below, the styling of\nerror messages and the mean value is altered.\n  ```Dart\n  import 'package:ansi_modifier/ansi_modifier.dart';\n  import 'package:benchmark_runner/benchmark_runner.dart';\n\n  void customColorProfile() {\n    ColorProfile.error = Ansi.red + Ansi.bold;\n    ColorProfile.mean = Ansi.green + Ansi.italic;\n  }\n\n  void main(List\u003cString\u003e args) {\n    // Call function to apply the new custom color profile.\n    customProfile();\n  }\n  ```\n\n- When running **asynchronous** benchmarks, the scores are printed in order of\ncompletion. The print the scores in sequential order (as they are listed in the\nbenchmark executable) it is required to *await* the completion\nof the async benchmark functions and\nthe enclosing group.\n\n## Score Sampling\n\nIn order to calculate benchmark score statistics a sample of scores is\nrequired. The question is how to generate the score sample while minimizing\nsystematic errors (like overheads) and keeping the\nbenchmark run times within acceptable limits.\n\nTo estimate the benchmark score the functions [`warmup`][warmup]\nor [`warmupAsync`][warmupAsync] are run for 200 milliseconds.\n\n### 1. Default Sampling Method\nThe graph below shows the sample size (orange curve) as calculated by the function\n[`BenchmarkHelper.sampleSize`][sampleSize].\nThe green curve shows the lower limit of the total microbenchmark duration and\nrepresents the value: `clockTicks * sampleSize * innerIterations`.\n\n![Sample Size](https://raw.githubusercontent.com/simphotonics/benchmark_runner/main/images/sample_size.png)\n\nFor short run times below 100000 clock ticks each sample score is generated\nusing the functions [`measure`][measure] or the equivalent asynchronous method [`measureAsync`][measureAsync].\nThe parameter\n`ticks` used when calling the functions [`measure`][measure] and\n[`measureAsync`][measureAsync] is chosen such that the benchmark score is\naveraged over (see the cyan curve in the graph above):\n* ticks \u003c 1000 =\u003e 200 runs,\n* 1000 \u003c ticks \u003c 1e4 =\u003e 200 ... 100 runs (exponentialy interpolated),\n* 1e4 \u003c ticks \u003c 1e5 =\u003e 100 ... 20 runs (exponentially interpolated),\n* ticks \u003e 1e5 =\u003e No preliminary averaging of sample scores.\n\n### 2. Custom Sampling Method\nTo amend the score sampling process the static function\n[`BenchmarkHelper.sampleSize`][sampleSize] can be replaced with a custom function:\n```Dart\nBenchmarkHelper.sampleSize = (int clockTicks) {\n  return (outer: 100, inner: 1)\n}\n```\nTo restore the default score sampling settings use:\n```Dart\nBenchmarkHelper.sampleSize = BenchmarkHelper.sampleSizeDefault;\n```\n----\nThe graph shown above may be re-generated using the custom `sampleSize`\nfunction by copying and amending the file `gnuplot/sample_size.dart`\nand using the command:\n```Console\ndart sample_size.dart\n```\nThe command above lauches a process and runs a [`gnuplot`][gnuplot] script.\nFor this reason, the program [`gnuplot`][gnuplot] must be installed (with\nthe `qt` terminal enabled).\n\n\n## Contributions\n\nHelp and enhancement requests are welcome. Please file requests via the [issue\ntracker][tracker].\n\nThe To-Do list currently includes:\n* Add tests.\n\n* Add color profiles optimized for terminals with light background color.\n\n* Improve the way benchmark score samples are generated.\n\n## Features and bugs\n\nPlease file feature requests and bugs at the [issue tracker][tracker].\n\n[tracker]: https://github.com/simphotonics/benchmark_runner/issues\n\n[asyncBenchmark]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/asyncBenchmark.html\n\n[asyncGroup]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/asyncGroup.html\n\n[benchmark_harness]: https://pub.dev/packages/benchmark_harness\n\n[benchmark_runner]: https://pub.dev/packages/benchmark_runner\n\n[benchmark]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/benchmark.html\n\n[ColorProfile]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/ColorProfile.html\n\n[gnuplot]: https://sourceforge.net/projects/gnuplot/\n\n[group]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/group.html\n\n[measure]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/measure.html\n\n[measureAsync]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/measureAsync.html\n\n[sampleSize]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/sampleSize.html\n\n[warmup]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/warmup.html\n\n[warmupAsync]: https://pub.dev/documentation/benchmark_runner/latest/benchmark_runner/BenchmarkHelper/warmupAsync.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimphotonics%2Fbenchmark_runner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimphotonics%2Fbenchmark_runner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimphotonics%2Fbenchmark_runner/lists"}