{"id":13466404,"url":"https://github.com/bencheeorg/benchee","last_synced_at":"2025-05-13T00:18:07.923Z","repository":{"id":39898080,"uuid":"59428665","full_name":"bencheeorg/benchee","owner":"bencheeorg","description":"Easy and extensible benchmarking in Elixir providing you with lots of statistics!","archived":false,"fork":false,"pushed_at":"2025-05-05T16:01:56.000Z","size":1821,"stargazers_count":1456,"open_issues_count":24,"forks_count":68,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-05-13T00:17:39.023Z","etag":null,"topics":["benchmarking","elixir","extensible","graphs","microbenchmarks","statistics"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bencheeorg.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-05-22T18:48:43.000Z","updated_at":"2025-05-06T20:05:15.000Z","dependencies_parsed_at":"2023-02-09T13:31:21.266Z","dependency_job_id":"e84d53aa-4a1a-4247-afe9-d987e6619ada","html_url":"https://github.com/bencheeorg/benchee","commit_stats":{"total_commits":1082,"total_committers":37,"mean_commits":"29.243243243243242","dds":"0.24584103512014788","last_synced_commit":"d1228710390ee9d871a7b31e8e458a4b334c7790"},"previous_names":["pragtob/benchee"],"tags_count":25,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bencheeorg%2Fbenchee","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bencheeorg%2Fbenchee/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bencheeorg%2Fbenchee/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bencheeorg%2Fbenchee/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bencheeorg","download_url":"https://codeload.github.com/bencheeorg/benchee/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253843225,"owners_count":21972874,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","elixir","extensible","graphs","microbenchmarks","statistics"],"created_at":"2024-07-31T15:00:43.605Z","updated_at":"2025-05-13T00:18:07.899Z","avatar_url":"https://github.com/bencheeorg.png","language":"Elixir","readme":"# Benchee [![Hex Version](https://img.shields.io/hexpm/v/benchee.svg)](https://hex.pm/packages/benchee) [![Hex Docs](https://img.shields.io/badge/docs-hexpm-blue.svg)](https://hexdocs.pm/benchee/) [![CI](https://github.com/bencheeorg/benchee/workflows/CI/badge.svg)](https://github.com/bencheeorg/benchee/actions?query=branch%3Amain) [![Coverage Status](https://coveralls.io/repos/github/bencheeorg/benchee/badge.svg?branch=main)](https://coveralls.io/github/bencheeorg/benchee?branch=main) [![Total Download](https://img.shields.io/hexpm/dt/benchee.svg)](https://hex.pm/packages/benchee) [![License](https://img.shields.io/hexpm/l/benchee.svg)](https://github.com/bencheeorg/benchee/blob/main/LICENSE)\n\nLibrary for easy and nice (micro) benchmarking in Elixir. Benchee allows you to compare the performance of different pieces of code at a glance. It is also versatile and extensible, relying only on functions. There are also a bunch of [plugins](#plugins) to draw pretty graphs and more!\n\nBenchee runs each of your functions for a given amount of time after an initial warmup, it then measures their run time and optionally memory consumption. It then shows different statistical values like average, standard deviation etc. See [features](#features).\n\nBenchee has a nice and concise main interface, its behavior can be altered through lots of [configuration options](#configuration):\n\n```elixir\nlist = Enum.to_list(1..10_000)\nmap_fun = fn i -\u003e [i, i * i] end\n\nBenchee.run(\n  %{\n    \"flat_map\" =\u003e fn -\u003e Enum.flat_map(list, map_fun) end,\n    \"map.flatten\" =\u003e fn -\u003e list |\u003e Enum.map(map_fun) |\u003e List.flatten() end\n  },\n  time: 10,\n  memory_time: 2\n)\n```\n\nProduces the following output on the console:\n\n```\nOperating System: Linux\nCPU Information: AMD Ryzen 9 5900X 12-Core Processor\nNumber of Available Cores: 24\nAvailable memory: 31.25 GB\nElixir 1.16.0-rc.1\nErlang 26.1.2\nJIT enabled: true\n\nBenchmark suite executing with the following configuration:\nwarmup: 2 s\ntime: 10 s\nmemory time: 2 s\nreduction time: 0 ns\nparallel: 1\ninputs: none specified\nEstimated total run time: 28 s\n\nBenchmarking flat_map ...\nBenchmarking map.flatten ...\nCalculating statistics...\nFormatting results...\n\nName                  ips        average  deviation         median         99th %\nflat_map           3.79 K      263.87 μs    ±15.49%      259.47 μs      329.29 μs\nmap.flatten        1.96 K      509.19 μs    ±51.36%      395.23 μs     1262.27 μs\n\nComparison:\nflat_map           3.79 K\nmap.flatten        1.96 K - 1.93x slower +245.32 μs\n\nMemory usage statistics:\n\nName           Memory usage\nflat_map             625 KB\nmap.flatten       781.25 KB - 1.25x memory usage +156.25 KB\n\n**All measurements for memory usage were the same**\n```\n\nThe aforementioned [plugins](#plugins) like [benchee_html](https://github.com/bencheeorg/benchee_html) make it possible to generate nice looking [HTML reports](http://www.pragtob.info/benchee/README/results.html), where individual graphs can also be exported as PNG images:\n\n![report](http://www.pragtob.info/benchee/images/report.png)\n\n## Table of Contents\n\n- [Table of Contents](#table-of-contents)\n- [Features](#features)\n  - [Statistics](#statistics)\n- [Installation](#installation)\n- [Usage](#usage)\n  - [Configuration](#configuration)\n  - [Metrics to measure](#metrics-to-measure)\n    - [Measuring time](#measuring-time)\n      - [A note on time measurement accuracy](#a-note-on-time-measurement-accuracy)\n    - [Measuring Memory Consumption](#measuring-memory-consumption)\n    - [Measuring Reductions](#measuring-reductions)\n  - [Inputs](#inputs)\n  - [Formatters](#formatters)\n    - [Console Formatter options](#console-formatter-options)\n  - [Profiling after a run](#profiling-after-a-run)\n  - [Saving, loading and comparing previous runs](#saving-loading-and-comparing-previous-runs)\n  - [Hooks (Setup, Teardown etc.)](#hooks-setup-teardown-etc)\n    - [Suite hooks](#suite-hooks)\n    - [Scenario hooks](#scenario-hooks)\n      - [What is a scenario?](#what-is-a-scenario)\n      - [before\\_scenario](#before_scenario)\n      - [after\\_scenario](#after_scenario)\n    - [Benchmarking function hooks](#benchmarking-function-hooks)\n      - [before\\_each](#before_each)\n      - [after\\_each](#after_each)\n    - [Hook arguments and return values](#hook-arguments-and-return-values)\n    - [Hook configuration: global versus local](#hook-configuration-global-versus-local)\n    - [When does a hook happen? (Complete Example)](#when-does-a-hook-happen-complete-example)\n  - [More verbose usage](#more-verbose-usage)\n  - [Usage from Erlang](#usage-from-erlang)\n- [Known Issues](#known-issues)\n- [Plugins](#plugins)\n- [Contributing ](#contributing-)\n- [Development](#development)\n- [Copyright and License](#copyright-and-license)\n\n## Features\n\n* first runs the functions for a given warmup time without recording the results, to simulate a _\"warm\"/running_ system\n* [measures memory usage](#measuring-memory-consumption)\n* provides you with lots of [statistics](#statistics)\n* plugin/extensible friendly architecture so you can use different formatters to display benchmarking results as [HTML, markdown, JSON and more](#plugins)\n* measure not only [time](#measuring-time), but also measure [memory](#measuring-memory-consumption) and [reductions](#measuring-reductions)\n* [profile](#profiling-after-a-run) right after benchmarking\n* as precise as it can get, measure with up to nanosecond precision (Operating System dependent)\n* nicely formatted console output with units scaled to appropriately (nanoseconds to minutes)\n* (optionally) measures the overhead of function calls so that the measured/reported times really are the execution time of _your_code_ without that overhead.\n* [hooks](#hooks-setup-teardown-etc) to execute something before/after a benchmarking invocation, without it impacting the measured time\n* execute benchmark jobs in parallel to gather more results in the same time, or simulate a system under load\n* well documented \u0026 well tested\n\n### Statistics\n\nProvides you with the following **statistical data**:\n\n* **average**   - average execution time/memory usage (the lower the better)\n* **ips**       - iterations per second, aka how often can the given function be executed within one second (the higher the better - good for graphing), only for run times\n* **deviation** - standard deviation (how much do the results vary), given as a percentage of the average (raw absolute values also available)\n* **median**    - when all measured values are sorted, this is the middle value. More stable than the average and somewhat more likely to be a typical value you see, for the most typical value see mode. (the lower the better)\n* **99th %**    - 99th percentile, 99% of all measured values are less than this - worst case performance-ish\n\nIn addition, you can optionally output an extended set of statistics:\n\n* **minimum**     - the smallest value measured for the job (fastest/least consumption)\n* **maximum**     - the biggest run time measured for the job (slowest/most consumption)\n* **sample size** - the number of measurements taken\n* **mode**        - the measured values that occur the most. Often one value, but can be multiple values if they occur exactly as often. If no value occurs at least twice, this value will be `nil`.\n\n## Installation\n\nAdd `:benchee` to your list of dependencies in `mix.exs`:\n\n```elixir\ndefp deps do\n  [\n    {:benchee, \"~\u003e 1.0\", only: :dev}\n  ]\nend\n```\n\nInstall via `mix deps.get` and then happy benchmarking as described in [Usage](#usage) :)\n\nElixir versions supported/tested against are 1.7+. It _should_ theoretically still work with 1.4, 1.5 \u0026 1.6 but we don't actively test/support it as installing them on current CI systems is not supported out of the box.\n\n## Usage\n\nAfter installing just write a little Elixir benchmarking script:\n\n```elixir\nlist = Enum.to_list(1..10_000)\nmap_fun = fn i -\u003e [i, i * i] end\n\nBenchee.run(\n  %{\n    \"flat_map\" =\u003e fn -\u003e Enum.flat_map(list, map_fun) end,\n    \"map.flatten\" =\u003e fn -\u003e list |\u003e Enum.map(map_fun) |\u003e List.flatten() end\n  }\n)\n```\n\n(names can also be specified as `:atoms` if you want to)\n\nThis produces the following output:\n\n```\ntobi@speedy:$ mix run samples/run_defaults.exs\nOperating System: Linux\nCPU Information: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz\nNumber of Available Cores: 8\nAvailable memory: 15.61 GB\nElixir 1.8.1\nErlang 21.2.7\n\nBenchmark suite executing with the following configuration:\nwarmup: 2 s\ntime: 5 s\nmemory time: 0 ns\nparallel: 1\ninputs: none specified\nEstimated total run time: 14 s\n\nBenchmarking flat_map...\nBenchmarking map.flatten...\n\nName                  ips        average  deviation         median         99th %\nflat_map           2.34 K      427.78 μs    ±16.02%      406.29 μs      743.01 μs\nmap.flatten        1.22 K      820.87 μs    ±19.29%      772.61 μs     1286.35 μs\n\nComparison:\nflat_map           2.34 K\nmap.flatten        1.22 K - 1.92x slower +393.09 μs\n```\n\nSee [Features](#features) for a description of the different statistical values and what they mean.\n\nIf you're looking to see how to make something specific work, please refer to the [samples](https://github.com/bencheeorg/benchee/tree/main/samples) directory. Also, especially when wanting to extend Benchee, check out the [hexdocs](https://hexdocs.pm/benchee/api-reference.html).\n\n### Configuration\n\nBenchee takes a wealth of configuration options, however those are entirely optional. Benchee ships with sensible defaults for all of these.\n\nIn the most common `Benchee.run/2` interface configuration options are passed as the second argument in the form of an keyword list:\n\n```elixir\nBenchee.run(%{\"some function\" =\u003e fn -\u003e magic end}, print: [benchmarking: false])\n```\n\nThe available options are the following (also documented in [hexdocs](https://hexdocs.pm/benchee/Benchee.Configuration.html#init/1)).\n\n* `warmup` - the time in seconds for which a benchmarking job should be run without measuring anything before \"real\" measurements start. This simulates a _\"warm\"/running_ system. Defaults to 2.\n* `time` - the time in seconds for how long each individual scenario (benchmarking job x input) should be run for measuring the execution times (run time performance). Defaults to 5.\n* `memory_time` - the time in seconds for how long [memory measurements](#measuring-memory-consumption) should be conducted. Defaults to 0 (turned off).\n* `reduction_time` - the time in seconds for how long [reductions are measured](#measuring-memory-reductions) should be conducted. Defaults to 0 (turned off).\n* `inputs` - a map or list of two element tuples. If a map, the keys are descriptive input names and values are the actual input values. If a list of tuples, the first element in each tuple is the input name, and the second element in each tuple is the actual input value. Your benchmarking jobs will then be run with each of these inputs. For this to work your benchmarking function gets the current input passed in as an argument into the function. Defaults to `nil`, aka no input specified and functions are called without an argument. See [Inputs](#inputs).\n* `formatters` - list of formatters either as a module implementing the formatter behaviour, a tuple of said module and options it should take or formatter functions. They are run when using `Benchee.run/2` or you can invoke them through `Benchee.Formatter.output/1`. Functions need to accept one argument (which is the benchmarking suite with all data) and then use that to produce output. Used for plugins \u0026 configuration. Also allows the configuration of the console formatter to print extended statistics. Defaults to the builtin console formatter `Benchee.Formatters.Console`. See [Formatters](#formatters).\n* `measure_function_call_overhead` - Measure how long an empty function call takes and deduct this from each measured run time. This overhead should be negligible for all but the most micro benchmarks. Defaults to false.\n* `pre_check` - whether or not to run each job with each input - including all given before or after scenario or each hooks - before the benchmarks are measured to ensure that your code executes without error. This can save time while developing your suites. Defaults to `false`. Possible values are:\n    * `false` - no pre check is run\n    * `true` - each scenario is run but the return value is ignored\n    * `:all_same` - raises if all scenarios are not returning the same value for\n    each input. This is useful when benchmarking alternative implementations of a\n    deterministic function.\n* `parallel` - the function of each benchmarking job will be executed in `parallel` number processes. If `parallel: 4` then 4 processes will be spawned that all execute the _same_ function for the given time. When `time` seconds have passed, 4 new processes will be spawned for the next scenario (meaning a new input or another function to be benchmarked). This gives you more data in the same time, but also puts load on the system interfering with benchmark results. For more on the pros and cons of parallel benchmarking [check the wiki](https://github.com/bencheeorg/benchee/wiki/Parallel-Benchmarking). Defaults to 1 (no parallel execution).\n* `save` - specify a `path` where to store the results of the current benchmarking suite, tagged with the specified `tag`. See [Saving \u0026 Loading](#saving-loading-and-comparing-previous-runs).\n* `load` - load saved suite or suites to compare your current benchmarks against. Can be a string or a list of strings or patterns. See [Saving \u0026 Loading](#saving-loading-and-comparing-previous-runs).\n* `print` - a map or keyword list from atoms to `true` or `false` to configure if the output identified by the atom will be printed during the standard Benchee benchmarking process. All options are enabled by default (true). Options are:\n  * `:benchmarking`  - print when Benchee starts benchmarking a new job (`Benchmarking name ...`) as well as when statistics are being calculated or formatting begins.\n  * `:configuration` - a summary of configured benchmarking options including estimated total run time is printed before benchmarking starts\n  * `:fast_warning` - warnings are displayed if functions are executed too fast leading to inaccurate measures\n* `:unit_scaling` - the strategy for choosing a unit for durations,\n  counts \u0026 memory measurements. May or may not be implemented by a given formatter (The console formatter implements it).\n  When scaling a value, Benchee finds the \"best fit\"\n  unit (the largest unit for which the result is at least 1). For example,\n  1_200_000 scales to `1.2 M`, while `800_000` scales to `800 K`. The\n  `unit_scaling` strategy determines how Benchee chooses the best fit unit for\n  an entire list of values, when the individual values in the list may have\n  different best fit units. There are four strategies, defaulting to `:best`:\n    * `:best`     - the most frequent best fit unit will be used, a tie\n    will result in the larger unit being selected.\n    * `:largest`  - the largest best fit unit will be used\n    * `:smallest` - the smallest best fit unit will be used\n    * `:none`     - no unit scaling will occur.\n* `:before_scenario`/`after_scenario`/`before_each`/`after_each` - read up on them in the [hooks section](#hooks-setup-teardown-etc)\n* `profile_after` - accepts any of the following options:\n    * a boolean   - `true` will enable profiling with the default profiler (`:eprof`) and `false` will disable profiling. Defaults to `false`.\n    * a profiler  - either as a tuple of `{profiler, opts}` (e.g., `{:fprof, [sort: :own]}`) or just the profiler (e.g., `:fprof`),\n    which is equivalent to `{profiler, []}`. The accepted built-in profilers are:\n    [`:cprof`](https://hexdocs.pm/mix/Mix.Tasks.Profile.Cprof.html),\n    [`:eprof`](https://hexdocs.pm/mix/Mix.Tasks.Profile.Eprof.html)\n    [`:fprof`](https://hexdocs.pm/mix/Mix.Tasks.Profile.Fprof.html) and\n    [`:tprof`](https://hexdocs.pm/mix/Mix.Tasks.Profile.Tprof.html) (requires elixir \u003e= 1.17 and erlang \u003e= 27).\n\n### Metrics to measure\n\nBenchee can't only measure [execution time](#measuring-time), but also [memory consumption](#measuring-memory-consumption) and [reductions](#measuring-reductions)!\n\nYou can measure one of these metrics, or all at the same time. The choice is up to you. Warmup will only occur once though, the time for measuring the metrics are governed by `time`, `memory_time` and `reduction_time` configuration values respectively.\n\nBy default only execution time is measured, memory and reductions need to be opted in by specifying a non 0 time amount.\n\n#### Measuring time\n\nThis is the default, which you'll most likely want to use as you want to measure how fast your system processes something or responds to a request. Benchee does its best to measure time as accurately and as scoped to your function as possible.\n\n```elixir\nBenchee.run(\n  %{\n    \"something_great\" =\u003e fn -\u003e cool_stuff end\n  },\n  warmup: 1,\n  time: 5,\n  memory_time: 2,\n  reduction_time: 2\n)\n```\n\n##### A note on time measurement accuracy\n\nFrom system to system the resolution of the clock [can vary](https://www.erlang.org/doc/apps/erts/time_correction.html).\n\nGenerally speaking we have seen accuracies down to 1 nanosecond on Linux and ~1 microsecond onb both OSX and Windows. We have also seen accuracy as low as 10ms on Windows in a CI environment.\nThese numbers are not a limitation of Benchee, but of the Operating System (or at the very least how erlang makes it available).\n\nIf your benchmark takes 100s of microseconds this likely has no/little impact, but **if you want to do extremely nano benchmarks we recommend doing them on Linux**.\n\nSo, what happens if a function executes too fast for Benchee to measure? If Benchee recognizes this (not always possible) it will automatically take the function and repeat it multiple times. This way a function that takes 2 microseconds will take ~20 microseconds and differences can be measured even with microsecond accuracy. Naturally, this comes at several disadvantages:\n\n* The looping/repetition code is now measured alongside the function\n* essentially every single measurement is now an average across 10 runs making lots of statistics less meaningful\n\nBenchee will print a big warning when this happens.\n#### Measuring Memory Consumption\n\nStarting with version 0.13, users can now get measurements of how much memory their benchmarked scenarios use. The measurement is **limited to the process that Benchee executes your provided code in** - i.e. other processes (like worker pools)/the whole BEAM isn't taken into account.\n\nThis measurement is **not** the actual effect on the size of the BEAM VM size, but the total amount of memory that was allocated during the execution of a given scenario. This includes all memory that was garbage collected during the execution of that scenario.\n\nThis measurement of memory does not affect the measurement of run times.\n\nIn cases where all measurements of memory consumption are identical, which happens very frequently, the full statistics will be omitted from the standard console formatter. If your function is deterministic, this should always be the case. Only in functions with some amount of randomness will there be variation in memory usage.\n\nMemory measurement is disabled by default, you can choose to enable it by passing `memory_time: your_seconds` option to `Benchee.run/2`:\n\n```elixir\nBenchee.run(\n  %{\n    \"something_great\" =\u003e fn -\u003e cool_stuff end\n  },\n  memory_time: 2\n)\n```\n\nMemory time can be specified separately as it will often be constant - so it might not need as much measuring time.\n\nA full example, including an example of the console output, can be found\n[here](samples/measure_memory.exs).\n\n#### Measuring Reductions\n\nStarting in versions 1.1.0 Benchee can measure reductions - but what are reductions?\n\nIn short, it's not very well defined but a \"unit of work\". The BEAM uses them to keep track of how long a process has run. As [the Beam Book puts it as follows](https://blog.stenmans.org/theBeamBook/#_scheduling_non_preemptive_reduction_counting):\n\n\u003eBEAM solves this by keeping track of how long a process has been running. This is done by counting reductions. The term originally comes from the mathematical term beta-reduction used in lambda calculus.\n\u003e\n\u003eThe definition of a reduction in BEAM is not very specific, but we can see it as a small piece of work, which shouldn’t take too long. Each function call is counted as a reduction. BEAM does a test upon entry to each function to check whether the process has used up all its reductions or not. If there are reductions left the function is executed otherwise the process is suspended.\n\nNow, why would you want to measure this? Well, apart from BIFs \u0026 NIFs, which are not accurately tracked through this, it gives an impression of how much work the BEAM is doing. And the great thing is, this is independent of the load the system is under as well as the hardware. Hence, it gives you a way to check performance that is less volatile so suitable for CI for instance.\n\nIt can slightly differ between elixir \u0026 erlang versions, though.\n\n**Like memory measurements, this only tracks reductions directly in the function given to benchee, not processes spawned by it or other processes it uses.**\n\nReduction measurement is disabled by default, you can choose to enable it by passing `reduction_time: your_seconds` option to `Benchee.run/2`:\n\n```elixir\nBenchee.run(\n  %{\n    \"something_great\" =\u003e fn -\u003e cool_stuff end\n  },\n  reduction_time: 2\n)\n```\n\nAlso like memory measurements, reduction measurements will often be constant unless something changes about the execution of the benchmarking function.\n\n### Inputs\n\n`:inputs` is a very useful configuration that allows you to run the same benchmarking jobs with different inputs. We call this combination a _\"scenario\"_. You specify the inputs as either a map from name (String or atom) to the actual input value or a list of tuples where the first element in each tuple is the name and the second element in the tuple is the value.\n\nWhy do this? Functions can have different performance characteristics on differently shaped inputs - be that structure or input size. One of such cases is comparing tail-recursive and body-recursive implementations of `map`. More information in the [repository with the benchmark](https://github.com/PragTob/elixir_playground/blob/main/bench/tco_blog_post_focussed_inputs.exs) and the [blog post](https://pragtob.wordpress.com/2016/06/16/tail-call-optimization-in-elixir-erlang-not-as-efficient-and-important-as-you-probably-think/).\n\nAs a little sample:\n\n```elixir\nmap_fun = fn i -\u003e [i, i * i] end\n\nBenchee.run(\n  %{\n    \"flat_map\" =\u003e fn input -\u003e Enum.flat_map(input, map_fun) end,\n    \"map.flatten\" =\u003e fn input -\u003e input |\u003e Enum.map(map_fun) |\u003e List.flatten() end\n  },\n  inputs: %{\n    \"Small\" =\u003e Enum.to_list(1..1_000),\n    \"Medium\" =\u003e Enum.to_list(1..10_000),\n    \"Bigger\" =\u003e Enum.to_list(1..100_000)\n  }\n)\n```\n\nThis means each function will be benchmarked with each input that is specified in the inputs. Then you'll get the output divided by input so you can see which function is fastest for which input, like so:\n\n```\ntobi@speedy:~/github/benchee(readme-overhaul)$ mix run samples/multiple_inputs.exs\n\n(... general information ...)\n\n##### With input Bigger #####\nName                  ips        average  deviation         median         99th %\nflat_map           150.81        6.63 ms    ±12.65%        6.57 ms        8.74 ms\nmap.flatten        114.05        8.77 ms    ±16.22%        8.42 ms       12.76 ms\n\nComparison:\nflat_map           150.81\nmap.flatten        114.05 - 1.32x slower +2.14 ms\n\n##### With input Medium #####\nName                  ips        average  deviation         median         99th %\nflat_map           2.28 K      437.80 μs    ±10.72%      425.63 μs      725.09 μs\nmap.flatten        1.78 K      561.18 μs     ±5.55%      553.98 μs      675.98 μs\n\nComparison:\nflat_map           2.28 K\nmap.flatten        1.78 K - 1.28x slower +123.37 μs\n\n##### With input Small #####\nName                  ips        average  deviation         median         99th %\nflat_map          26.31 K       38.01 μs    ±15.47%       36.69 μs       67.08 μs\nmap.flatten       18.65 K       53.61 μs    ±11.32%       52.79 μs       70.17 μs\n\nComparison:\nflat_map          26.31 K\nmap.flatten       18.65 K - 1.41x slower +15.61 μs\n```\n\nTherefore, we **highly recommend** using this feature and checking different realistically structured and sized inputs for the functions you benchmark!\n\n### Formatters\n\nAmong all the configuration options, one that you probably want to use are the formatters. They format and print out the results of the benchmarking suite.\n\nThe `:formatters` option is specified a list of:\n* modules implementing the `Benchee.Formatter` behaviour\n* a tuple of a module specified above and options for it `{module, options}`\n* functions that take one argument (the benchmarking suite with all its results) and then do whatever you want them to\n\nSo if you are using the [HTML plugin](https://github.com/bencheeorg/benchee_html) and you want to run both the console formatter and the HTML formatter this looks like this (after you installed it of course):\n\n```elixir\nlist = Enum.to_list(1..10_000)\nmap_fun = fn i -\u003e [i, i * i] end\n\nBenchee.run(\n  %{\n    \"flat_map\"    =\u003e fn -\u003e Enum.flat_map(list, map_fun) end,\n    \"map.flatten\" =\u003e fn -\u003e list |\u003e Enum.map(map_fun) |\u003e List.flatten end\n  },\n  formatters: [\n    {Benchee.Formatters.HTML, file: \"samples_output/my.html\"},\n    Benchee.Formatters.Console\n  ]\n)\n```\n\n#### Console Formatter options\n\nThe console formatter supports 2 configuration options:\n\n  * `:comparison` - if the comparison of the different benchmarking jobs (x times slower than) is shown. Enabled by default.\n  * `extended_statistics` - display more statistics, aka `minimum`, `maximum`, `sample_size` and `mode`. Disabled by default.\n\nSo if you want to see more statistics you simple pass `extended_statistics: true` to the console formatter:\n\n```elixir\nlist = Enum.to_list(1..10_000)\nmap_fun = fn i -\u003e [i, i * i] end\n\nBenchee.run(\n  %{\n    \"flat_map\" =\u003e fn -\u003e Enum.flat_map(list, map_fun) end,\n    \"map.flatten\" =\u003e fn -\u003e list |\u003e Enum.map(map_fun) |\u003e List.flatten() end\n  },\n  time: 10,\n  formatters: [{Benchee.Formatters.Console, extended_statistics: true}]\n)\n```\n\nWhich produces:\n\n```\n(... normal output ...)\n\nExtended statistics:\n\nName                minimum        maximum    sample size                     mode\nflat_map          345.43 μs     1195.89 μs        23.73 K                405.64 μs\nmap.flatten       522.99 μs     2105.56 μs        12.03 K     767.83 μs, 768.44 μs\n```\n\n(Btw. notice how the modes of both are much closer and for `map.flatten` much less than the average of `766.99`, see `samples/run_extended_statistics`)\n\n### Profiling after a run\n\nOften time when benchmarking, you want to improve performance. However, Benchee only tells you how fast something is, it doesn't tell you what part of your code is slow. This is where profiling comes in.\n\nBenchee offers you the `profile_after` option to run a profiler of your choice after a benchmarking run to see what's slow. This will run every scenario once.\n\nBy default it will run the `:eprof` profiler, different profilers with different options can be used - see [configuration](#configuration).\n\n```elixir\nlist = Enum.to_list(1..10_000)\nmap_fun = fn i -\u003e [i, i * i] end\n\nBenchee.run(\n  %{\n    \"flat_map\" =\u003e fn -\u003e Enum.flat_map(list, map_fun) end,\n    \"map.flatten\" =\u003e fn -\u003e list |\u003e Enum.map(map_fun) |\u003e List.flatten() end\n  },\n  profile_after: true\n)\n```\n\nAfter the normal benchmarking, this results prints profiles like:\n\n```\nProfiling flat_map with eprof...\nWarmup...\n\n\nProfile results of #PID\u003c0.1885.0\u003e\n#                                               CALLS     % TIME µS/CALL\nTotal                                           30004 100.0 5665    0.19\nEnum.flat_map/2                                     1  0.00    0    0.00\nanonymous fn/2 in :elixir_compiler_1.__FILE__/1     1  0.00    0    0.00\n:erlang.apply/2                                     1  0.04    2    2.00\n:erlang.++/2                                    10000 18.38 1041    0.10\nanonymous fn/1 in :elixir_compiler_1.__FILE__/1 10000 28.12 1593    0.16\nEnum.flat_map_list/2                            10001 53.47 3029    0.30\n\nProfile done over 6 matching functions\n\nProfiling map.flatten with eprof...\nWarmup...\n\n\nProfile results of #PID\u003c0.1887.0\u003e\n#                                               CALLS     % TIME µS/CALL\nTotal                                           60007 100.0 8649    0.14\nEnum.map/2                                          1  0.00    0    0.00\nanonymous fn/2 in :elixir_compiler_1.__FILE__/1     1  0.00    0    0.00\nList.flatten/1                                      1  0.00    0    0.00\n:lists.flatten/1                                    1  0.00    0    0.00\n:erlang.apply/2                                     1  0.02    2    2.00\nanonymous fn/1 in :elixir_compiler_1.__FILE__/1 10000 16.35 1414    0.14\nEnum.\"-map/2-lists^map/1-0-\"/2                  10001 26.38 2282    0.23\n:lists.do_flatten/2                             40001 57.24 4951    0.12\n```\n\n**Note about after_each hooks:** `after_each` hooks currently don't work when profiling a function, as they are not passed the return value of the function after the profiling run. It's already fixed on the elixir side and is waiting for release, likely in 1.14. It should then just work.\n\n### Saving, loading and comparing previous runs\n\nBenchee can store the results of previous runs in a file and then load them again to compare them. For example this is useful to compare what was recorded on the main branch against a branch with performance improvements. You may also use this to benchmark across different exlixir/erlang versions.\n\n**Saving** is done through the `:save` configuration option. You can specify a `:path` where results are saved, or you can use the default option of`\"benchmark.benchee\"` if you don't pass a `:path`. You can also pass a `:tag` option which annotates these results (for instance with a branch name or elixir version number). The default option for the `:tag` is a timestamp of when the benchmark was run.\n\n**Loading** is done through the `:load` option specifying a path to the file to load (for instance `\"benchmark.benchee\"`). You can also specify multiple files to load through a list of paths (`[\"my.benchee\", \"main_save.benchee\"]`) - each one of those can also be a glob expression to match even more files glob (`\"save_number*.benchee\"`).\nIn the more verbose API loading is triggered via `Benchee.load/1`.\nIf all you want to do is to use `:load` without running any benchmarks then you can use `Benchee.report/1` which will take a normal configuration with a `:load` key and just format the loaded saved results with the given formatters.\n\n```elixir\nBenchee.run(\n  %{\n    \"something_great\" =\u003e fn -\u003e cool_stuff end\n  },\n  save: [path: \"save.benchee\", tag: \"first-try\"]\n)\n\nBenchee.report(load: \"save.benchee\")\n```\n\n### Hooks (Setup, Teardown etc.)\n\nMost of the time, it's best to keep your benchmarks as simple as possible: plain old immutable functions work best. But sometimes you need other things to happen. When you want to add before or after hooks to your benchmarks, we've got you covered! Before you dig into this section though remember one thing: **you usually don't need hooks!**\n\nBenchee has three types of hooks:\n\n* [Suite hooks](#suite-hooks)\n* [Scenario hooks](#scenario-hooks)\n* [Benchmarking function hooks](#benchmarking-function-hooks)\n\n\nOf course, **hooks are not included in the measurements**. So they are there especially if you want to do something and want it to **not be included in the measurements**. Sadly there is the notable exception of _too_fast_functions_ (the ones that execute faster than we can measure in [_native_ resolution](#a-note-on-time-measurement-accuracy)). As we need to measure their repeated invocations to get halfway good measurements `before_each` and `after_each` hooks are included there. However, to the best of our knowledge this should only ever happen on Windows (because of the bad run time measurement accuracy).\n\n#### Suite hooks\n\nIt is very easy in Benchee to do setup and teardown for the whole benchmarking suite (think: \"before all\" and \"after all\"). As Benchee is just plain old functions, just do your setup and teardown before/after you call Benchee:\n\n```elixir\nyour_setup()\n\nBenchee.run(%{\"Awesome stuff\" =\u003e fn -\u003e magic end})\n\nyour_teardown()\n```\n\n_When might this be useful?_\n\n* Seeding the database with data to be used by all benchmarking functions\n* Starting/shutting down a server, process, other dependencies\n\n#### Scenario hooks\n\nFor the following discussions, it's important to know what benchee considers a \"benchmarking scenario\".\n\n##### What is a scenario?\n\nA scenario is the combination of one benchmarking function and one input. So, given this benchmark:\n\n```elixir\nBenchee.run(\n  %{\n    \"foo\" =\u003e fn input -\u003e ... end,\n    \"bar\" =\u003e fn input -\u003e ... end\n  },\n  inputs: %{\n    \"input 1\" =\u003e 1,\n    \"input 2\" =\u003e 2\n  }\n)\n```\n\nthere are 4 scenarios:\n\n1. foo with input 1\n2. foo with input 2\n3. bar with input 1\n4. bar with input 2\n\nA scenario includes warmup and actual run time (+ other measurements like memory).\n\n##### before_scenario\n\nIs executed before every [scenario](#what-is-a-scenario) that it applies to (see [hook configuration](#hook-configuration-global-versus-local)). `before_scenario` hooks take the input of the scenario as an argument.\n\nSince the return value of a `before_scenario` becomes the input for next steps (see [hook arguments and return values](#hook-arguments-and-return-values)),  there usually are 3 kinds of before scenarios:\n\n* you just want to invoke a side effect: in that case return the given input unchanged\n* you want to alter the given input: in that case alter the given input\n* you want to keep the given input but add some other data: in that case return a tuple like `{original_input, new_fancy_data}`\n\nFor before scenario hooks, the _global_ hook is invoked first, then the _local_ (see [when does a hook happen?](#when-does-a-hook-happen-complete-example)).\n\nUsage:\n\n```elixir\nBenchee.run(\n  %{\n    \"foo\" =\u003e\n      {\n        fn {input, resource} -\u003e foo(input, resource) end,\n        before_scenario: fn {input, resource} -\u003e\n          resource = alter_resource(resource)\n          {input, resource}\n        end\n      },\n    \"bar\" =\u003e\n      fn {input, resource} -\u003e bar(input, resource) end\n  },\n  inputs: %{\"input 1\" =\u003e 1},\n  before_scenario: fn input -\u003e\n    resource = start_expensive_resource()\n    {input, resource}\n  end\n)\n```\n\n_When might this be useful?_\n\n* Starting a process to be used in your scenario\n* Recording the PID of `self()` for use in your benchmark (each scenario is executed in its own process, so scenario PIDs aren't available in functions running before the suite)\n* Clearing the cache before a scenario\n\n##### after_scenario\n\nIs executed after a scenario has completed. After scenario hooks receive the return value of the last `before_scenario` that ran as an argument. The return value is discarded (see [hook arguments and return values](#hook-arguments-and-return-values)).\n\nFor after scenario hooks, the _local_ hook is invoked first, then the _global_ (see [when does a hook happen?](#when-does-a-hook-happen-complete-example)).\n\nUsage:\n\n```elixir\nBenchee.run(\n  %{\n    \"bar\" =\u003e fn -\u003e bar() end\n  },\n  after_scenario: fn _input -\u003e bust_my_cache() end\n)\n```\n\n_When might this be useful?_\n\n* Busting caches after a scenario completed\n* Deleting all the records in the database that the scenario just created\n* Terminating whatever was setup by `before_scenario`\n\n#### Benchmarking function hooks\n\nYou can also schedule hooks to run before and after each invocation of a benchmarking function.\n\n##### before_each\n\nIs executed before each invocation of the benchmarking function (before every measurement). Before each hooks receive the return value of their `before_scenario` hook as their argument. The return value of a before each hook becomes the input to the benchmarking function (see [hook arguments and return values](#hook-arguments-and-return-values)).\n\nFor before each hooks, the _global_ hook is invoked first, then the _local_ (see [when does a hook happen?](#when-does-a-hook-happen-complete-example)).\n\nUsage:\n\n```elixir\nBenchee.run(\n  %{\n    \"bar\" =\u003e fn record -\u003e bar(record) end\n  },\n  inputs: %{ \"record id\" =\u003e 1},\n  before_each: fn input -\u003e get_from_db(input) end\n)\n```\n\n_When might this be useful?_\n\n* Retrieving a record from the database and passing it on to the benchmarking function to do _something(tm)_ without the retrieval from the database adding to the benchmark measurement\n* Busting caches so that all measurements are taken in an uncached state\n* Picking a random value from a collection and passing it to the benchmarking function for measuring performance with a wider spread of values\n* You could also use this to benchmark with random data generated by tools like [StreamData](https://github.com/whatyouhide/stream_data), Devon shows how it's done [here](https://devonestes.com/benchmarking-with-stream-data)\n\n##### after_each\n\nIs executed directly after the invocation of the benchmarking function. After each hooks receive the return value of the benchmarking function as their argument. The return value is discarded (see [hook arguments and return values](#hook-arguments-and-return-values)).\n\nFor after each hooks, the _local_ hook is invoked first, then the _global_ (see [when does a hook happen?](#when-does-a-hook-happen-complete-example)).\n\nUsage:\n\n```elixir\nBenchee.run(\n  %{\n    \"bar\" =\u003e fn input -\u003e bar(input) end\n  },\n  inputs: %{ \"input 1\" =\u003e 1},\n  after_each: fn result -\u003e assert result == 42 end\n)\n```\n\n_When might this be useful?_\n\n* Running assertions that whatever your benchmarking function returns is truly what it should be (i.e. all \"contestants\" work as expected)\n* Busting caches/terminating processes setup in `before_each` or elsewhere\n* Deleting files created by the benchmarking function\n\n#### Hook arguments and return values\n\nBefore hooks form a chain, where the return value of the previous hook becomes the argument for the next one. The first defined `before` hook receives the scenario input as an argument, and returns a value that becomes the argument of the next in the chain. The benchmarking function receives the value of the last `before` hook as its argument (or the scenario input if there are no `before` hooks).\n\nAfter hooks do not form a chain, and their return values are discarded. An `after_each` hook receives the return value of the benchmarking function as its argument. An `after_scenario` function receives the return value of the last `before_scenario` that ran (or the scenario's input if there is no `before_scenario` hook).\n\nIf you haven't defined any inputs, the hook chain is started with the special input argument returned by `Benchee.Benchmark.no_input()`.\n\n#### Hook configuration: global versus local\n\nHooks can be defined either _globally_ as part of the configuration or _locally_ for a specific benchmarking function. Global hooks will be executed for every scenario in the suite. Local hooks will only be executed for scenarios involving that benchmarking function.\n\n**Global hooks**\n\nGlobal hooks are specified as part of the general benchee configuration:\n\n```elixir\nBenchee.run(\n  %{\n    \"foo\" =\u003e fn input -\u003e ... end,\n    \"bar\" =\u003e fn input -\u003e ... end\n  },\n  inputs: %{\n    \"input 1\" =\u003e 1,\n    \"input 2\" =\u003e 2,\n  },\n  before_scenario: fn input -\u003e ... end\n)\n```\n\nHere the `before_scenario` function will be executed for all 4 scenarios present in this benchmarking suite.\n\n**Local hooks**\n\nLocal hooks are defined alongside the benchmarking function. To define a local hook, pass a tuple in the initial map, instead of just a single function. The benchmarking function comes first, followed by a keyword list specifying the hooks to run:\n\n```elixir\nBenchee.run(\n  %{\n    \"foo\" =\u003e {fn input -\u003e ... end, before_scenario: fn input -\u003e ... end},\n    \"bar\" =\u003e fn input -\u003e ... end\n  },\n  inputs: %{\n    \"input 1\" =\u003e 1,\n    \"input 2\" =\u003e 2\n  }\n)\n```\n\nHere `before_scenario` is only run for the 2 scenarios associated with `\"foo\"`, i.e. foo with input 1 and foo with input 2. It is _not_ run for any `\"bar\"` benchmarks.\n\n\n#### When does a hook happen? (Complete Example)\n\nYes the whole hooks system is quite a lot to take in. Here is an overview showing the order of hook execution, along with the argument each hook receives (see [hook arguments and return values](#hook-arguments-and-return-values)). The guiding principle whether _local_ or _global_ is run first is that _local_ always executes closer to the benchmarking function.\n\nGiven the following code:\n\n```elixir\n\nsuite_set_up()\n\nBenchee.run(\n  %{\n    \"foo\" =\u003e\n      {\n        fn input -\u003e foo(input) end,\n        before_scenario: fn input -\u003e\n          local_before_scenario(input)\n          input + 1\n        end,\n        before_each: fn input -\u003e\n          local_before_each(input)\n          input + 1\n        end,\n        after_each: fn value -\u003e\n          local_after_each(value)\n        end,\n        after_scenario: fn input -\u003e\n          local_after_scenario(input)\n        end\n      },\n    \"bar\" =\u003e\n      fn input -\u003e bar(input) end\n  },\n  inputs: %{\"input 1\" =\u003e 1},\n  before_scenario: fn input -\u003e\n    global_before_scenario(input)\n    input + 1\n  end,\n  before_each: fn input -\u003e\n    global_before_each(input)\n    input + 1\n  end,\n  after_each: fn value -\u003e\n    global_after_each(value)\n  end,\n  after_scenario: fn input -\u003e\n    global_after_scenario(input)\n  end\n)\n\nsuite_tear_down()\n```\n\nKeeping in mind that the order of foo and bar could be different, here is how the hooks are called:\n\n```\nsuite_set_up\n\n# starting with the foo scenario\nglobal_before_scenario(1)\nlocal_before_scenario(2) # as before_scenario incremented it\n\nglobal_before_each(3)\nlocal_before_each(4)\nfoo(5) # let's say foo(5) returns 6\nlocal_after_each(6)\nglobal_after_each(6)\n\nglobal_before_each(3)\nlocal_before_each(4)\nfoo(5) # let's say foo(5) returns 6\nlocal_after_each(6)\nglobal_after_each(6)\n\n# ... this block is repeated as many times as benchee has time\n\nlocal_after_scenario(3)\nglobal_after_scenario(3)\n\n# now it's time for the bar scenario, it has no hooks specified for itself\n# so only the global hooks are run\n\nglobal_before_scenario(1)\n\nglobal_before_each(2)\nbar(3) # let's say foo(3) returns 4\nglobal_after_each(4)\n\nglobal_before_each(2)\nbar(3) # let's say foo(3) returns 4\nglobal_after_each(4)\n\n# ... this block is repeated as many times as benchee has time\n\nglobal_after_scenario(2)\n\nsuite_tear_down\n```\n\n### More verbose usage\n\nIt is important to note that the benchmarking code shown in the beginning is the convenience interface. The same benchmark in its more verbose form looks like this:\n\n```elixir\nlist = Enum.to_list(1..10_000)\nmap_fun = fn i -\u003e [i, i * i] end\n\n[time: 3]\n|\u003e Benchee.init()\n|\u003e Benchee.system()\n|\u003e Benchee.benchmark(\"flat_map\", fn -\u003e Enum.flat_map(list, map_fun) end)\n|\u003e Benchee.benchmark(\n  \"map.flatten\",\n  fn -\u003e list |\u003e Enum.map(map_fun) |\u003e List.flatten() end\n)\n|\u003e Benchee.collect()\n|\u003e Benchee.statistics()\n|\u003e Benchee.relative_statistics()\n|\u003e Benchee.Formatter.output(Benchee.Formatters.Console)\n# Instead of the last call you could also just use Benchee.Formatter.output()\n# to just output all configured formatters\n```\n\nThis is a take on the _functional transformation_ of data applied to benchmarks:\n\n1. Configure the benchmarking suite to be run\n2. Gather System data\n3. Define the functions to be benchmarked\n4. Run benchmarks with the given configuration gathering raw run times per function\n5. Calculate statistics based on the raw run times\n6. Calculate statistics between the scenarios (faster/slower...)\n7. Format the statistics in a suitable way and print them out\n\nThis is also part of the **official API** and allows for more **fine grained control**. (It's also what Benchee does internally when you use `Benchee.run/2`).\n\nDo you just want to have all the raw run times? Just work with the result of `Benchee.collect/1`! Just want to have the calculated statistics and use your own formatting? Grab the result of `Benchee.statistics/1`! Or, maybe you want to write to a file or send an HTTP post to some online service? Just grab the complete suite after statistics were generated.\n\nIt also allows you to alter behaviour, normally `Benchee.load/1` is called right before the formatters so that neither the benchmarks are run again or statistics are computed again. However, you might want to run the benchmarks again or recompute the statistics. Then you can call `Benchee.load/1` right at the start.\n\nThis way Benchee should be flexible enough to suit your needs and be extended at will. Have a look at the [available plugins](#plugins).\n\n### Usage from Erlang\n\nBefore you dig deep into this, it is inherently easier to setup a small elixir project, add a dependency to your erlang project and then run the benchmarks from elixir. The reason is easy - mix knows about rebar3 and knows how to work with it. The reverse isn't true so the road ahead is somewhat bumpy.\n\nThere is an [example project](https://github.com/bencheeorg/benchee_erlang_try) to check out.\n\nYou can use the [rebar3_elixir_compile](https://github.com/barrel-db/rebar3_elixir_compile) plugin. In your `rebar.config` you can do the following which should get you started:\n\n```erlang\ndeps, [\n  {benchee, {elixir, \"benchee\", \"0.9.0\"}}\n]}.\n\n{plugins, [\n    { rebar3_elixir_compile, \".*\", {git, \"https://github.com/barrel-db/rebar3_elixir_compile.git\", {branch, \"main\"}}}\n]}.\n\n{provider_hooks, [\n  {pre, [{compile, {ex, compile}}]},\n  {pre, [{release, {ex, compile}}]}\n]}.\n\n{elixir_opts,\n  [\n    {env, dev}\n  ]\n}.\n```\n\nThen benchee already provides a `:benchee` interface for erlang compatibility which you can use. Sadly couldn't get it to work in an escript yet.\n\nYou can then invoke it like this (for instance):\n\n```\ntobi@comfy ~/github/benchee_erlang_try $ rebar3 shell\n===\u003e dependencies etc.\nErlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]\n\nEshell V7.3  (abort with ^G)\n1\u003e benchee:run(#{myFunc =\u003e fun() -\u003e lists:sort([8, 2, 3, 4, 2, 1, 3, 4, 9, 10, 11, 12, 13, 20, 1000, -4, -5]) end}, [{warmup, 0}, {time, 2}]).\nOperating System: Linux\nCPU Information: Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz\nNumber of Available Cores: 4\nAvailable memory: 8.05 GB\nElixir 1.3.4\nErlang 18.3\nBenchmark suite executing with the following configuration:\nwarmup: 0.0 μs\ntime: 2 s\nparallel: 1\ninputs: none specified\nEstimated total run time: 2 s\n\n\nBenchmarking myFunc...\n\nName             ips        average  deviation         median\nmyFunc      289.71 K        3.45 μs   ±250.31%           3 μs\n```\n\nThis doesn't seem to be too reliable right now, so suggestions and input are very welcome :)\n\n## Known Issues\n\nThere is a known issue affecting elixir versions from 1.14.0 to 1.16.0-rc.0: Optimizations (SSA and bool passes, [see the original change](https://github.com/elixir-lang/elixir/pull/11420)) had been disabled affecting the performance of functions defined directly in the top level (i.e. outside of any module). The issue was fixed by re-enabling the optimization in [1.16.0-rc.1](https://github.com/elixir-lang/elixir/blob/v1.16/CHANGELOG.md#v1160-rc1-2023-12-12).\nThe issue is best show-cased by the following benchmark where we'd expect ~equal results:\n\n```elixir\nlist = Enum.to_list(1..10_000)\n\ndefmodule Compiled do\n  def comprehension(list) do\n    for x \u003c- list, rem(x, 2) == 1, do: x + 1\n  end\nend\n\nBenchee.run(%{\n  \"module (optimized)\" =\u003e fn -\u003e Compiled.comprehension(list) end,\n  \"top_level (non-optimized)\" =\u003e fn -\u003e for x \u003c- list, rem(x, 2) == 1, do: x + 1 end\n})\n```\n\nWhich yields ~these results on an affected elixir version:\n\n```\nComparison:\nmodule (optimized)              18.24 K\ntop_level (non-optimized)       11.91 K - 1.53x slower +29.14 μs\n```\n\nSo, how do you fix it/make sure a benchmark you ran is not affected? All of these work:\n\n* benchmark on an unaffected/fixed version of elixir (\u003c= 1.13.4 or \u003e= 1.16.0-rc.1)\n* put the code you want to benchmark into a module (just like it is done in `Compiled` in the example above)\n* you can also invoke benchee from within a module, such as:\n\n```elixir\ndefmodule Compiled do\n  def comprehension(list) do\n    for x \u003c- list, rem(x, 2) == 1, do: x + 1\n  end\nend\n\ndefmodule MyBenchmark do\n  def run do\n    list = Enum.to_list(1..10_000)\n\n    Benchee.run(%{\n      \"module (optimized)\" =\u003e fn -\u003e Compiled.comprehension(list) end,\n      \"top_level (non-optimized)\" =\u003e fn -\u003e for x \u003c- list, rem(x, 2) == 1, do: x + 1 end\n    })\n  end\nend\n\nMyBenchmark.run()\n```\n\nAlso note that even if all your examples are top level functions you should still follow these tips (on affected elixir versions), as the missing optimization might affect them differently. Further note, that even though your examples use top level functions they _may_ not be affected, as the specific disabled optimization may not impact them. Better safe than sorry though :)\n\n## Plugins\n\nBenchee only has small runtime dependencies that were initially extracted from it. Further functionality is provided through plugins that then pull in dependencies, such as HTML generation and CSV export. They help provide excellent visualization or interoperability.\n\n* [benchee_html](//github.com/bencheeorg/benchee_html) - generate HTML including a data table and many different graphs with the possibility to export individual graphs as PNG :)\n* [benchee_csv](//github.com/bencheeorg/benchee_csv) - generate CSV from your benchee benchmark results so you can import them into your favorite spreadsheet tool and make fancy graphs\n* [benchee_json](//github.com/bencheeorg/benchee_json) - export suite results as JSON to feed anywhere or feed it to your JavaScript and make magic happen :)\n* [benchee_markdown](//github.com/hrzndhrn/benchee_markdown) - write markdown files containing your benchmarking results\n\nWith the HTML plugin for instance you can get fancy graphs like this boxplot:\n\n![boxplot](http://www.pragtob.info/benchee/images/boxplot.png)\n\nOf course there also are normal bar charts including standard deviation:\n\n![flat_map_ips](http://www.pragtob.info/benchee/images/flat_map_ips.png)\n\n## Contributing [![Open Source Helpers](https://www.codetriage.com/pragtob/benchee/badges/users.svg)](https://www.codetriage.com/pragtob/benchee)\n\nContributions to Benchee are **very welcome**! Bug reports, documentation, spelling corrections, whole features, feature ideas, bugfixes, new plugins, fancy graphics... all of those (and probably more) are much appreciated contributions!\n\nKeep in mind that the [plugins](#plugins) live in their own repositories with their own issue tracker and they also like to get contributions :)\n\nPlease respect the [Code of Conduct](//github.com/bencheeorg/benchee/blob/main/CODE_OF_CONDUCT.md).\n\nIn addition to contributing code, you can help to triage issues. This can include reproducing bug reports, or asking for vital information such as version numbers or reproduction instructions. If you would like to start triaging issues, one easy way to get started is to [subscribe to pragtob/benchee on CodeTriage](https://www.codetriage.com/pragtob/benchee).\n\nYou can also look directly at the [open issues](https://github.com/bencheeorg/benchee/issues). There are `help wanted` and `good first issue` labels - those are meant as guidance, of course other issues can be tackled :)\n\nA couple of (hopefully) helpful points:\n\n* Feel free to ask for help and guidance on an issue/PR (\"How can I implement this?\", \"How could I test this?\", ...)\n* Feel free to open early/not yet complete pull requests to get some early feedback\n* When in doubt if something is a good idea open an issue first to discuss it\n* In case I don't respond feel free to bump the issue/PR or ping me in other places\n\nIf you're on the [elixir-lang slack](https://elixir-lang.slack.com) also feel free to drop by in `#benchee` and say hi!\n\n## Development\n\nNote that if the change includes adding new statistics you might need to introduce them to our statistics library [Statistex](https://github.com/bencheeorg/statistex) first!\n\nFor an overview of benchee's architecture, check out [architecture](./architecture/README.md)!\n\n* `mix deps.get` to install dependencies\n* `mix test` to run tests\n* `mix dialyzer` to run dialyzer for type checking, might take a while on the first invocation (try building plts first with `mix dialyzer --plt`)\n* `mix credo` to find code style problems\n* or run `mix guard` to run all of them continuously on file change\n\n\n## Copyright and License\n\nCopyright (c) 2016 Tobias Pfeiffer\n\nThis work is free. You can redistribute it and/or modify it under the\nterms of the MIT License. See the [LICENSE.md](./LICENSE.md) file for more details.\n","funding_links":[],"categories":["Elixir"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbencheeorg%2Fbenchee","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbencheeorg%2Fbenchee","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbencheeorg%2Fbenchee/lists"}