{"id":16661408,"url":"https://github.com/bodigrim/tasty-bench","last_synced_at":"2025-04-09T13:05:56.556Z","repository":{"id":39904115,"uuid":"330506059","full_name":"Bodigrim/tasty-bench","owner":"Bodigrim","description":"Featherlight benchmark framework, drop-in replacement for criterion and gauge.","archived":false,"fork":false,"pushed_at":"2024-07-28T17:07:14.000Z","size":221,"stargazers_count":80,"open_issues_count":2,"forks_count":11,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-13T10:34:48.595Z","etag":null,"topics":["benchmark","benchmark-framework","benchmark-suite","benchmarking","benchmarking-suite","benchmarks","criterion","gauge","haskell","performance-analysis","performance-measurement","performance-testing"],"latest_commit_sha":null,"homepage":"https://hackage.haskell.org/package/tasty-bench","language":"Haskell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Bodigrim.png","metadata":{"files":{"readme":"README.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-01-17T23:16:38.000Z","updated_at":"2024-09-14T16:38:48.000Z","dependencies_parsed_at":"2023-09-22T01:19:20.175Z","dependency_job_id":"351d9c29-dab0-4753-80a3-5a2501c89385","html_url":"https://github.com/Bodigrim/tasty-bench","commit_stats":{"total_commits":168,"total_committers":5,"mean_commits":33.6,"dds":0.04166666666666663,"last_synced_commit":"920ca44cd5c45274a8e9ea23b38883bc6278343a"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Ftasty-bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Ftasty-bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Ftasty-bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Bodigrim%2Ftasty-bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Bodigrim","download_url":"https://codeload.github.com/Bodigrim/tasty-bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248045231,"owners_count":21038553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","benchmark-framework","benchmark-suite","benchmarking","benchmarking-suite","benchmarks","criterion","gauge","haskell","performance-analysis","performance-measurement","performance-testing"],"created_at":"2024-10-12T10:34:52.948Z","updated_at":"2025-04-09T13:05:56.534Z","avatar_url":"https://github.com/Bodigrim.png","language":"Haskell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tasty-bench [![Hackage](http://img.shields.io/hackage/v/tasty-bench.svg)](https://hackage.haskell.org/package/tasty-bench) [![Stackage LTS](http://stackage.org/package/tasty-bench/badge/lts)](http://stackage.org/lts/package/tasty-bench) [![Stackage Nightly](http://stackage.org/package/tasty-bench/badge/nightly)](http://stackage.org/nightly/package/tasty-bench)\n\nFeatherlight benchmark framework (only one file!) for performance measurement\nwith API mimicking [`criterion`](http://hackage.haskell.org/package/criterion)\nand [`gauge`](http://hackage.haskell.org/package/gauge).\nA prominent feature is built-in comparison against previous runs\nand between benchmarks.\n\n\u003c!-- MarkdownTOC autolink=\"true\" --\u003e\n\n- [How lightweight is it?](#how-lightweight-is-it)\n- [How is it possible?](#how-is-it-possible)\n- [How to switch?](#how-to-switch)\n- [How to write a benchmark?](#how-to-write-a-benchmark)\n- [How to read results?](#how-to-read-results)\n- [Wall-clock time vs. CPU time](#wall-clock-time-vs-cpu-time)\n- [Statistical model](#statistical-model)\n- [Memory usage](#memory-usage)\n- [Combining tests and benchmarks](#combining-tests-and-benchmarks)\n- [Troubleshooting](#troubleshooting)\n- [Isolating interfering benchmarks](#isolating-interfering-benchmarks)\n- [Comparison against baseline](#comparison-against-baseline)\n- [Comparison between benchmarks](#comparison-between-benchmarks)\n- [Plotting results](#plotting-results)\n- [Build flags](#build-flags)\n- [Command-line options](#command-line-options)\n- [Custom command-line options](#custom-command-line-options)\n\n\u003c!-- /MarkdownTOC --\u003e\n\n## How lightweight is it?\n\nThere is only one source file `Test.Tasty.Bench` and no non-boot dependencies\nexcept [`tasty`](http://hackage.haskell.org/package/tasty).\nSo if you already depend on `tasty` for a test suite, there\nis nothing else to install.\n\nCompare this to `criterion` (10+ modules, 50+ dependencies) and `gauge` (40+ modules, depends on `basement` and `vector`). A build on a clean machine is up to 16x\nfaster than `criterion` and up to 4x faster than `gauge`. A build without dependencies\nis up to 6x faster than `criterion` and up to 8x faster than `gauge`.\n\n`tasty-bench` is a native Haskell library and works everywhere, where GHC\ndoes, including WASM. We support a full range of architectures (`i386`, `amd64`, `armhf`,\n`arm64`, `ppc64le`, `s390x`) and operating systems (Linux, Windows, macOS,\nFreeBSD, OpenBSD, NetBSD), plus any GHC from 8.0 to 9.10\n(and earlier releases stretch back to GHC 7.0).\n\n## How is it possible?\n\nOur benchmarks are literally regular `tasty` tests, so we can leverage all existing\nmachinery for command-line options, resource management, structuring,\nlisting and filtering benchmarks, running and reporting results. It also means\nthat `tasty-bench` can be used in conjunction with other `tasty` ingredients.\n\nUnlike `criterion` and `gauge` we use a very simple statistical model described below.\nThis is arguably a questionable choice, but it works pretty well in practice.\nA rare developer is sufficiently well-versed in probability theory\nto make sense and use of all numbers generated by `criterion`.\n\n## How to switch?\n\n[Cabal mixins](https://cabal.readthedocs.io/en/3.4/cabal-package.html#pkg-field-mixins)\nallow to taste `tasty-bench` instead of `criterion` or `gauge`\nwithout changing a single line of code:\n\n```cabal\ncabal-version: 2.0\n\nbenchmark foo\n  ...\n  build-depends:\n    tasty-bench\n  mixins:\n    tasty-bench (Test.Tasty.Bench as Criterion, Test.Tasty.Bench as Criterion.Main, Test.Tasty.Bench as Gauge, Test.Tasty.Bench as Gauge.Main)\n```\n\nThis works vice versa as well: if you use `tasty-bench`, but at some point\nneed a more comprehensive statistical analysis,\nit is easy to switch temporarily back to `criterion`.\n\n## How to write a benchmark?\n\nBenchmarks are declared in a separate section of `cabal` file:\n\n```cabal\ncabal-version:   2.0\nname:            bench-fibo\nversion:         0.0\nbuild-type:      Simple\nsynopsis:        Example of a benchmark\n\nbenchmark bench-fibo\n  main-is:       BenchFibo.hs\n  type:          exitcode-stdio-1.0\n  build-depends: base, tasty-bench\n  ghc-options:   \"-with-rtsopts=-A32m\"\n  if impl(ghc \u003e= 8.6)\n    ghc-options: -fproc-alignment=64\n```\n\nAnd here is `BenchFibo.hs`:\n\n```haskell\nimport Test.Tasty.Bench\n\nfibo :: Int -\u003e Integer\nfibo n = if n \u003c 2 then toInteger n else fibo (n - 1) + fibo (n - 2)\n\nmain :: IO ()\nmain = defaultMain\n  [ bgroup \"Fibonacci numbers\"\n    [ bench \"fifth\"     $ nf fibo  5\n    , bench \"tenth\"     $ nf fibo 10\n    , bench \"twentieth\" $ nf fibo 20\n    ]\n  ]\n```\n\nSince `tasty-bench` provides an API compatible with `criterion`,\none can refer to [its documentation](http://www.serpentine.com/criterion/tutorial.html#how-to-write-a-benchmark-suite) for more examples.\n\n## How to read results?\n\nRunning the example above (`cabal bench` or `stack bench`)\nresults in the following output:\n\n```\nAll\n  Fibonacci numbers\n    fifth:     OK (2.13s)\n       63 ns ± 3.4 ns\n    tenth:     OK (1.71s)\n      809 ns ±  73 ns\n    twentieth: OK (3.39s)\n      104 μs ± 4.9 μs\n\nAll 3 tests passed (7.25s)\n```\n\nThe output says that, for instance, the first benchmark was repeatedly\nexecuted for 2.13 seconds (wall-clock time), its predicted mean CPU time was\n63 nanoseconds and means of individual samples do not often diverge from it\nfurther than ±3.4 nanoseconds (double standard deviation). Take standard\ndeviation numbers with a grain of salt; there are lies, damned lies, and\nstatistics.\n\n## Wall-clock time vs. CPU time\n\nWhat time are we talking about?\nBoth `criterion` and `gauge` by default report wall-clock time, which is\naffected by any other application which runs concurrently.\nIdeally benchmarks are executed on a dedicated server without any other load,\nbut — let's face the truth — most of developers run benchmarks\non a laptop with a hundred other services and a window manager, and\nwatch videos while waiting for benchmarks to finish. That's the cause\nof a notorious \"variance introduced by outliers: 88% (severely inflated)\" warning.\n\nTo alleviate this issue `tasty-bench` measures CPU time by `getCPUTime`\ninstead of wall-clock time by default.\nIt does not provide a perfect isolation from other processes (e. g.,\nif CPU cache is spoiled by others, populating data back from RAM\nis your burden), but is a bit more stable.\n\nCaveat: this means that for multithreaded algorithms\n`tasty-bench` reports total elapsed CPU time across all cores, while\n`criterion` and `gauge` print maximum of core's wall-clock time.\nIt also means that by default `tasty-bench` does not measure time spent out of process,\ne. g., calls to other executables. To work around this limitation\nuse `--time-mode` command-line option or set it locally via `TimeMode` option.\n\n## Statistical model\n\nHere is a procedure used by `tasty-bench` to measure execution time:\n\n1. Set $n \\leftarrow 1$.\n2. Measure execution time $t_n$ of $n$ iterations\n   and execution time $t_{2n}$ of $2n$ iterations.\n3. Find $t$ which minimizes deviation of $(nt,2nt)$ from $(t_n,t_{2n})$,\n   namely $t \\leftarrow (t_n + 2t_{2n}) / 5n$.\n4. If deviation is small enough (see `--stdev` below)\n   or time is running out soon (see `--timeout` below),\n   return $t$ as a mean execution time.\n5. Otherwise set $n \\leftarrow 2n$ and jump back to Step 2.\n\nThis is roughly similar to the linear regression approach which `criterion` takes,\nbut we fit only two last points. This allows us to simplify away all heavy-weight\nstatistical analysis. More importantly, earlier measurements,\nwhich are presumably shorter and noisier, do not affect overall result.\nThis is in contrast to `criterion`, which fits all measurements and\nis biased to use more data points corresponding to shorter runs\n(it employs $n \\leftarrow 1.05n$ progression).\n\nMean time and its deviation does not say much about the\ndistribution of individual timings. E. g., imagine a computation which\n(according to a coarse system timer) takes either 0 ms or 1 ms with equal\nprobability. While one would be able to establish that its mean time is 0.5 ms\nwith a very small deviation, this does not imply that individual measurements\nare anywhere near 0.5 ms. Even assuming an infinite precision of a system\ntimer, the distribution of individual times is not known to be\n[normal](https://en.wikipedia.org/wiki/Normal_distribution).\n\nObligatory disclaimer: statistics is a tricky matter, there is no\none-size-fits-all approach.\nIn the absence of a good theory\nsimplistic approaches are as (un)sound as obscure ones.\nThose who seek statistical soundness should rather collect raw data\nand process it themselves using a proper statistical toolbox.\nData reported by `tasty-bench`\nis only of indicative and comparative significance.\n\n## Memory usage\n\nConfiguring RTS to collect GC statistics\n(e. g., via `cabal bench --benchmark-options '+RTS -T'`\nor `stack bench --ba '+RTS -T'`) enables `tasty-bench` to estimate and report\nmemory usage:\n\n```\nAll\n  Fibonacci numbers\n    fifth:     OK (2.13s)\n       63 ns ± 3.4 ns, 223 B  allocated,   0 B  copied, 2.0 MB peak memory\n    tenth:     OK (1.71s)\n      809 ns ±  73 ns, 2.3 KB allocated,   0 B  copied, 4.0 MB peak memory\n    twentieth: OK (3.39s)\n      104 μs ± 4.9 μs, 277 KB allocated,  59 B  copied, 5.0 MB peak memory\n\nAll 3 tests passed (7.25s)\n```\n\nThis data is reported as per [`GHC.Stats.RTSStats`](https://hackage.haskell.org/package/base/docs/GHC-Stats.html#t:RTSStats) fields:\n\n* `allocated_bytes`\n\n  Total size of data ever allocated since the start\n  of the benchmark iteration. Even if data was immediately\n  garbage collected and freed, it still counts.\n\n* `copied_bytes`\n\n  Total size of data ever copied by GC (because it was alive and kicking)\n  since the start of the benchmark iteration. Note that zero bytes often mean\n  that the benchmark was too short to trigger GC at all.\n\n* `max_mem_in_use_bytes`\n\n  Peak size of live data since the very start of the process.\n  This is a global metric, it cumulatively grows and does not say much\n  about individual benchmarks, but rather characterizes heap\n  environment in which they are executed.\n\n## Combining tests and benchmarks\n\nWhen optimizing an existing function, it is important to check that its\nobservable behavior remains unchanged. One can rebuild\nboth tests and benchmarks after each change, but it would be more convenient\nto run sanity checks within benchmark itself. Since our benchmarks\nare compatible with `tasty` tests, we can easily do so.\n\nImagine you come up with a faster function `myFibo` to generate Fibonacci numbers:\n\n```haskell\nimport Test.Tasty.Bench\nimport Test.Tasty.QuickCheck -- from tasty-quickcheck package\n\nfibo :: Int -\u003e Integer\nfibo n = if n \u003c 2 then toInteger n else fibo (n - 1) + fibo (n - 2)\n\nmyFibo :: Int -\u003e Integer\nmyFibo n = if n \u003c 3 then toInteger n else myFibo (n - 1) + myFibo (n - 2)\n\nmain :: IO ()\nmain = Test.Tasty.Bench.defaultMain -- not Test.Tasty.defaultMain\n  [ bench \"fibo   20\" $ nf fibo   20\n  , bench \"myFibo 20\" $ nf myFibo 20\n  , testProperty \"myFibo = fibo\" $ \\n -\u003e fibo n === myFibo n\n  ]\n```\n\nThis outputs:\n\n```\nAll\n  fibo   20:     OK (3.02s)\n    104 μs ± 4.9 μs\n  myFibo 20:     OK (1.99s)\n     71 μs ± 5.3 μs\n  myFibo = fibo: FAIL\n    *** Failed! Falsified (after 5 tests and 1 shrink):\n    2\n    1 /= 2\n    Use --quickcheck-replay=927711 to reproduce.\n\n1 out of 3 tests failed (5.03s)\n```\n\nWe see that `myFibo` is indeed significantly faster than `fibo`,\nbut unfortunately does not do the same thing. One should probably\nlook for another way to speed up generation of Fibonacci numbers.\n\n## Troubleshooting\n\n* If benchmarks take too long, set `--timeout` to limit execution time\n  of individual benchmarks, and `tasty-bench` will do its best to fit\n  into a given time frame. Without `--timeout` we rerun benchmarks until\n  achieving a target precision set by `--stdev`, which in a noisy environment\n  of a modern laptop with GUI may take a lot of time.\n\n  While `criterion` runs each benchmark at least for 5 seconds,\n  `tasty-bench` is happy to conclude earlier, if it does not compromise\n  the quality of results. In our experiments `tasty-bench` suites\n  tend to finish earlier, even if some individual benchmarks\n  take longer than with `criterion`.\n\n  A common source of noisiness is garbage collection. Setting a larger\n  allocation area (_nursery_) is often a good idea, either via\n  `cabal bench --benchmark-options '+RTS -A32m'` or `stack bench --ba '+RTS -A32m'`.\n  Alternatively bake it into\n  `cabal` file as `ghc-options: \"-with-rtsopts=-A32m\"`.\n\n* Never compile benchmarks with `-fstatic-argument-transformation`, because it\n  breaks a trick we use to force GHC into reevaluation of the same function application\n  over and over again.\n\n* If benchmark results look malformed like below, make sure that you are\n  invoking `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain`\n  (the difference is `consoleBenchReporter` vs. `consoleTestReporter`):\n\n  ```\n  All\n    fibo 20:       OK (1.46s)\n      WithLoHi (Estimate {estMean = Measurement {measTime = 41529118775, measAllocs = 0, measCopied = 0, measMaxMem = 0}, estStdev = 1595055320}) (-Infinity) Infinity\n  ```\n\n* If benchmarks fail with an error message\n\n  ```\n  Unhandled resource. Probably a bug in the runner you're using.\n  ```\n\n  or\n\n  ```\n  Unexpected state of the resource (NotCreated) in getResource. Report as a tasty bug.\n  ```\n\n  this is likely caused by `env` or `envWithCleanup` affecting benchmarks structure.\n  You can use `env` to read test data from `IO`, but not to read benchmark names\n  or affect their hierarchy in other way. This is a fundamental restriction of `tasty`\n  to list and filter benchmarks without launching missiles.\n\n  Strict pattern-matching on resource is also prohibited. For instance,\n  if it is a tuple, the second argument of `env` should use a lazy pattern match\n  `\\~(a, b) -\u003e ...`\n\n* If benchmarks fail with `Test dependencies form a loop`\n  or `Test dependencies have cycles`, this is likely\n  because of `bcompare`, which compares a benchmark with itself.\n  Locating a benchmark in a global environment may be tricky, please refer to\n  [`tasty` documentation](https://github.com/UnkindPartition/tasty#patterns) for details\n  and consider using `locateBenchmark`.\n\n* When seeing\n\n  ```\n  This benchmark takes more than 100 seconds. Consider setting --timeout, if this is unexpected (or to silence this warning).\n  ```\n\n  do follow the advice: abort benchmarks and pass `-t100` or similar. Unless you are\n  benchmarking a very computationally expensive function, a single benchmark should\n  stabilize after a couple of seconds. This warning is a sign that your environment\n  is too noisy, in which case `tasty-bench` will continue trying with exponentially\n  longer intervals, often unproductively.\n\n* The following error can be thrown when benchmarks are built with\n  `ghc-options: -threaded`:\n\n  ```\n  Benchmarks must not be run concurrently. Please pass -j1 and/or avoid +RTS -N.\n  ```\n\n  The underlying cause is that `tasty` runs tests concurrently, which is harmful\n  for reliable performance measurements. Make sure to use `tasty-bench \u003e= 0.3.4` and invoke\n  `Test.Tasty.Bench.defaultMain` and not `Test.Tasty.defaultMain`. Note that\n  `localOption (NumThreads 1)` quashes the warning, but does not eliminate the cause.\n\n* If benchmarks using GHC 9.4.4+ segfault on Windows, check that you\n  are not using non-moving garbage collector `--nonmoving-gc`. This is likely caused\n  by [GHC issue](https://gitlab.haskell.org/ghc/ghc/-/issues/23003).\n  Previous releases of `tasty-bench` recommended enabling `--nonmoving-gc`\n  to stabilise benchmarks, but it's discouraged now.\n\n* If you see\n\n  ```\n  \u003cstdout\u003e: commitBuffer: invalid argument (cannot encode character '\\177')\n  ```\n\n  or\n\n  ```\n  Uncaught exception ghc-internal:GHC.Internal.IO.Exception.IOException:\n  \u003cstdout\u003e: commitBuffer: invalid argument (cannot encode character '\\956')\n  ```\n\n  it means that your locale does not support UTF-8. `tasty-bench` makes an effort\n  to force locale to UTF-8, but sometimes, when benchmarks are a part of\n  a larger application, it's [impossible](https://gitlab.haskell.org/ghc/ghc/-/issues/23606)\n  to do so. In such case run `locale -a` to list available locales and set a UTF-8-capable\n  one (e. g., `export LANG=C.UTF-8`) before starting benchmarks.\n\n## Isolating interfering benchmarks\n\nOne difficulty of benchmarking in Haskell is that it is\nhard to isolate benchmarks so that they do not interfere.\nChanging the order of benchmarks or skipping some of them\nhas an effect on heap's layout and thus affects garbage collection.\nThis issue is well attested in\n[both](https://github.com/haskell/criterion/issues/166)\n[`criterion`](https://github.com/haskell/criterion/issues/60)\nand\n[`gauge`](https://github.com/vincenthz/hs-gauge/issues/2).\n\nUsually (but not always) skipping some benchmarks speeds up remaining ones.\nThat's because once a benchmark allocated heap which for some reason\nwas not promptly released afterwards (e. g., it forced a top-level thunk\nin an underlying library), all further benchmarks are slowed down\nby garbage collector processing this additional amount of live data\nover and over again.\n\nThere are several mitigation strategies. First of all, giving garbage collector\nmore breathing space by `+RTS -A32m` (or more) is often good enough.\n\nFurther, avoid using top-level bindings to store large test data. Once such thunks\nare forced, they remain allocated forever, which affects detrimentally subsequent\nunrelated benchmarks. Treat them as external data, supplied via `env`: instead of\n\n```haskell\nlargeData :: String\nlargeData = replicate 1000000 'a'\n\nmain :: IO ()\nmain = defaultMain\n  [ bench \"large\" $ nf length largeData, ... ]\n```\n\nuse\n\n```haskell\nimport Control.DeepSeq (force)\nimport Control.Exception (evaluate)\n\nmain :: IO ()\nmain = defaultMain\n  [ env (evaluate (force (replicate 1000000 'a'))) $ \\largeData -\u003e\n    bench \"large\" $ nf length largeData, ... ]\n```\n\nFinally, as an ultimate measure to reduce interference between benchmarks,\none can run each of them in a separate process. We do not quite recommend\nthis approach, but if you are desperate, here is how:\n\n```sh\ncabal run -v0 all:benches -- -l | sed -e 's/[\\\"]/\\\\\\\\\\\\\u0026/g' | while read -r name; do cabal run -v0 all:benches -- -p '$0 == \"'\"$name\"'\"'; done\n```\n\nThis assumes that there is a single benchmark suite in the project\nand that benchmark names do not contain newlines.\n\n## Comparison against baseline\n\nOne can compare benchmark results against an earlier run in an automatic way.\n\nWhen using this feature, it's especially important to compile benchmarks with\n`ghc-options: `[`-fproc-alignment`](https://downloads.haskell.org/ghc/latest/docs/users_guide/debugging.html#ghc-flag--fproc-alignment)`=64`, otherwise results could be skewed by\nintermittent changes in cache-line alignment.\n\nFirstly, run `tasty-bench` with `--csv FILE` key\nto dump results to `FILE` in CSV format\n(it could be a good idea to set smaller `--stdev`, if possible):\n\n```\nName,Mean (ps),2*Stdev (ps)\nAll.Fibonacci numbers.fifth,48453,4060\nAll.Fibonacci numbers.tenth,637152,46744\nAll.Fibonacci numbers.twentieth,81369531,3342646\n```\n\nNow modify implementation and rerun benchmarks\nwith `--baseline FILE` key. This produces a report as follows:\n\n```\nAll\n  Fibonacci numbers\n    fifth:     OK (0.44s)\n       53 ns ± 2.7 ns,  8% more than baseline\n    tenth:     OK (0.33s)\n      641 ns ±  59 ns,       same as baseline\n    twentieth: OK (0.36s)\n       77 μs ± 6.4 μs,  5% less than baseline\n\nAll 3 tests passed (1.50s)\n```\n\nYou can also fail benchmarks, which deviate too far from baseline, using\n`--fail-if-slower` and `--fail-if-faster` options. For example, setting both of them\nto 6 will fail the first benchmark above (because it is more than 6% slower),\nbut the last one still succeeds (even while it is measurably faster than baseline,\ndeviation is less than 6%). Consider also using `--hide-successes` to show\nonly problematic benchmarks, or even\n[`tasty-rerun`](http://hackage.haskell.org/package/tasty-rerun) package\nto focus on rerunning failing items only.\n\nIf you wish to compare two CSV reports non-interactively, here is a handy `awk` incantation:\n\n```sh\nawk 'BEGIN{FS=\",\";OFS=\",\";print \"Name,Old,New,Ratio\"}FNR==1{trueNF=NF;next}NF\u003ctrueNF{print \"Benchmark names should not contain newlines\";exit 1}FNR==NR{oldTime=$(NF-trueNF+2);NF-=trueNF-1;a[$0]=oldTime;next}{newTime=$(NF-trueNF+2);NF-=trueNF-1;if(a[$0]){print $0,a[$0],newTime,newTime/a[$0];gs+=log(newTime/a[$0]);gc++}}END{if(gc\u003e0)print \"Geometric mean,,\",exp(gs/gc)}' old.csv new.csv\n```\n\nA larger shell snippet to compare two `git` commits can be found in `compare_benches.sh`.\n\nNote that columns in CSV report are different from what `criterion` or `gauge`\nwould produce. If names do not contain commas, missing columns can be faked this way:\n\n```sh\nawk 'BEGIN{FS=\",\";OFS=\",\";print \"Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB\"}NR==1{trueNF=NF;next}NF\u003ctrueNF{print $0;next}{mean=$(NF-trueNF+2);stddev=$(NF-trueNF+3);NF-=trueNF-1;print $0,mean/1e12,mean/1e12,mean/1e12,stddev/2e12,stddev/2e12,stddev/2e12}'\n```\n\nTo fake `gauge` in `--csvraw` mode use\n\n```sh\nawk 'BEGIN{FS=\",\";OFS=\",\";print \"name,iters,time,cycles,cpuTime,utime,stime,maxrss,minflt,majflt,nvcsw,nivcsw,allocated,numGcs,bytesCopied,mutatorWallSeconds,mutatorCpuSeconds,gcWallSeconds,gcCpuSeconds\"}NR==1{trueNF=NF;next}NF\u003ctrueNF{print $0;next}{mean=$(NF-trueNF+2);fourth=$(NF-trueNF+4);fifth=$(NF-trueNF+5);sixth=$(NF-trueNF+6);NF-=trueNF-1;print $0,1,mean/1e12,0,mean/1e12,mean/1e12,0,sixth+0,0,0,0,0,fourth+0,0,fifth+0,0,0,0,0}'\n```\n\n## Comparison between benchmarks\n\nYou can also compare benchmarks to each other without any external tools,\nall in the comfort of your terminal.\n\n```haskell\nimport Test.Tasty.Bench\n\nfibo :: Int -\u003e Integer\nfibo n = if n \u003c 2 then toInteger n else fibo (n - 1) + fibo (n - 2)\n\nmain :: IO ()\nmain = defaultMain\n  [ bgroup \"Fibonacci numbers\"\n    [ bcompare \"tenth\"  $ bench \"fifth\"     $ nf fibo  5\n    ,                     bench \"tenth\"     $ nf fibo 10\n    , bcompare \"tenth\"  $ bench \"twentieth\" $ nf fibo 20\n    ]\n  ]\n```\n\nThis produces a report, comparing mean times of `fifth` and `twentieth` to `tenth`:\n\n```\nAll\n  Fibonacci numbers\n    fifth:     OK (16.56s)\n      121 ns ± 2.6 ns, 0.08x\n    tenth:     OK (6.84s)\n      1.6 μs ±  31 ns\n    twentieth: OK (6.96s)\n      203 μs ± 4.1 μs, 128.36x\n```\n\nTo locate a baseline benchmark in a larger suite use `locateBenchmark`.\n\nOne can leverage comparisons between benchmarks to implement portable performance\ntests, expressing properties like \"this algorithm must be at least twice faster\nthan that one\" or \"this operation should not be more than thrice slower than that\".\nThis can be achieved with `bcompareWithin`, which takes an acceptable interval\nof performance as an argument.\n\n## Plotting results\n\nUsers can dump results into CSV with `--csv FILE`\nand plot them using `gnuplot` or other software. But for convenience\nthere is also a built-in quick-and-dirty SVG plotting feature,\nwhich can be invoked by passing `--svg FILE`. Here is a sample of its output:\n\n![Plotting](https://hackage.haskell.org/package/tasty-bench/src/example.svg)\n\n## Build flags\n\nBuild flags are a brittle subject and users do not normally need to touch them.\n\n* If you find yourself in an environment, where `tasty` is not available and you\n  have access to boot packages only, you can still use `tasty-bench`! Just copy\n  `Test/Tasty/Bench.hs` to your project (imagine it like a header-only C library).\n  It will provide you with functions to build `Benchmarkable` and run them manually\n  via `measureCpuTime`. This mode of operation can be also configured\n  by disabling Cabal flag `tasty`.\n\n## Command-line options\n\nUse `--help` to list all command-line options.\n\n* `-p`, `--pattern`\n\n  This is a standard `tasty` option, which allows filtering benchmarks\n  by a pattern or `awk` expression. Please refer to\n  [`tasty` documentation](https://github.com/UnkindPartition/tasty#patterns)\n  for details.\n\n* `-t`, `--timeout`\n\n  This is a standard `tasty` option, setting timeout for individual benchmarks\n  in seconds. Use it when benchmarks tend to take too long: `tasty-bench` will make\n  an effort to report results (even if of subpar quality) before timeout. Setting\n  timeout too tight (insufficient for at least three iterations)\n  will result in a benchmark failure. One can adjust it locally for a group\n  of benchmarks, e. g., `localOption (mkTimeout 100000000)` for 100 seconds.\n\n* `--stdev`\n\n  Target relative standard deviation of measurements in percents (5% by default).\n  Large values correspond to fast and loose benchmarks, and small ones to long and precise.\n  It can also be adjusted locally for a group of benchmarks,\n  e. g., `localOption (RelStDev 0.02)`.\n  If benchmarking takes far too long, consider setting `--timeout`,\n  which will interrupt benchmarks, potentially before reaching the target deviation.\n\n* `--csv`\n\n  File to write results in CSV format.\n\n* `--baseline`\n\n  File to read baseline results in CSV format (as produced by `--csv`).\n\n* `--fail-if-slower`, `--fail-if-faster`\n\n  Upper bounds of acceptable slow down / speed up in percents. If a benchmark is unacceptably slower / faster than baseline (see `--baseline`),\n  it will be reported as failed. Can be used in conjunction with\n  a standard `tasty` option `--hide-successes` to show only problematic benchmarks.\n  Both options can be adjusted locally for a group of benchmarks,\n  e. g., `localOption (FailIfSlower 0.10)`.\n\n* `--svg`\n\n  File to plot results in SVG format.\n\n* `--time-mode`\n\n  Whether to measure CPU time (`cpu`, default) or wall-clock time (`wall`).\n\n* `+RTS -T`\n\n  Estimate and report memory usage.\n\n## Custom command-line options\n\nAs usual with `tasty`, it is easy to extend benchmarks with custom command-line options.\nHere is an example:\n\n```haskell\nimport Data.Proxy\nimport Test.Tasty.Bench\nimport Test.Tasty.Ingredients.Basic\nimport Test.Tasty.Options\nimport Test.Tasty.Runners\n\nnewtype RandomSeed = RandomSeed Int\n\ninstance IsOption RandomSeed where\n  defaultValue = RandomSeed 42\n  parseValue = fmap RandomSeed . safeRead\n  optionName = pure \"seed\"\n  optionHelp = pure \"Random seed used in benchmarks\"\n\nmain :: IO ()\nmain = do\n  let customOpts  = [Option (Proxy :: Proxy RandomSeed)]\n      ingredients = includingOptions customOpts : benchIngredients\n  opts \u003c- parseOptions ingredients benchmarks\n  let RandomSeed seed = lookupOption opts\n  defaultMainWithIngredients ingredients benchmarks\n\nbenchmarks :: Benchmark\nbenchmarks = bgroup \"All\" []\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbodigrim%2Ftasty-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbodigrim%2Ftasty-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbodigrim%2Ftasty-bench/lists"}