{"id":13507470,"url":"https://github.com/joekain/bmark","last_synced_at":"2025-10-21T17:44:51.985Z","repository":{"id":27369707,"uuid":"30845215","full_name":"joekain/bmark","owner":"joekain","description":"A benchmarking tool for Elixir","archived":false,"fork":false,"pushed_at":"2018-03-11T03:29:21.000Z","size":59,"stargazers_count":71,"open_issues_count":5,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-21T12:55:41.646Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/joekain.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-15T22:46:14.000Z","updated_at":"2024-07-08T14:21:09.000Z","dependencies_parsed_at":"2022-07-24T15:01:55.821Z","dependency_job_id":null,"html_url":"https://github.com/joekain/bmark","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joekain%2Fbmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joekain%2Fbmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joekain%2Fbmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/joekain%2Fbmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/joekain","download_url":"https://codeload.github.com/joekain/bmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246296361,"owners_count":20754625,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T02:00:34.464Z","updated_at":"2025-10-21T17:44:46.947Z","avatar_url":"https://github.com/joekain.png","language":"Elixir","funding_links":[],"categories":["Benchmarking"],"sub_categories":[],"readme":"# Bmark\n\n[![Build Status](https://travis-ci.org/joekain/bmark.svg?branch=master)](https://travis-ci.org/joekain/bmark)  [![Inline docs](http://inch-ci.org/github/joekain/bmark.svg?branch=master)](http://inch-ci.org/github/joekain/bmark)  [![Package](https://img.shields.io/hexpm/v/bmark.svg)](https://hex.pm/packages/bmark)\n\nBmark is a benchmarking tool for Elixir.  It allows easy benchmarking of Elixir functions.  It also supports comparing sets of benchmarking results.\n\nComparing benchmarking results is a topic that I have struggled with for years.  I run a benchmark several times and get varying results.  Then, I make a change to my program and I want to decide if the change causes an improvement in the benchmark score.  I rerun the benchmark several times and again get varying results.  How do I compare these results?  I can compare average score, but is that accurate?  How do I tell if the mean of the second run is large enough to be meaningful?  How do I know if it is \"in the noise?\"\n\nBmark answers this questions using statistical hypothesis testing.  Given two sets of benchmark runs, bmark can show:\n\n    RunA:                                 RunB:\n    24274268                              6426990\n    24563751                              6416149\n    24492221                              6507946\n    24516553                              6453309\n    24335224                              6491314\n    24158102                              6405073\n    24357174                              6504260\n    24213098                              6449789\n    24466586                              6532929\n    24289248                              6509800\n\n    24366622.5 -\u003e 6469755.9 (-73.45%) with p \u003c 0.0005\n    t = 391.56626146910503, 18 degrees of freedom\n\nThis shows that RunA ran in an average of 24366622.5 ms and RunB ran in an average of 6469755.9 ms and that the runtime improved by 73.45% which is statistically meaningful with a confidence level of 99.95%.\n\n## Usage\n\nAdd Bmark as a depdency in your mix.exs file:\n\n    def deps do\n      [ {:bmark, \"~\u003e 1.0.0\"} ]\n    end\n\n### Writing Benchmarks\n\nTo create a benchmark with bmark, create a file ending in `_bmark.ex`.  Put the file in a directory called `bmark`.  Alltogether, that should look like this\n\n    Project Root\n    +-- bmark\n    |   +-- example_bmark.ex\n    +-- lib\n    |   +-- your_project_files\n    +-- mix.exs\n\nIn `example_bmark.ex` you should include a module and benchmark function created by using `bmark`\nlike this:\n\n```elixir\ndefmodule Example do\n  use Bmark\n\n  bmark :runner do\n    IO.puts \":runner test is running\"\n  end\n\n  bmark :benchmark_with_runs, runs: 5 do\n    IO.puts \"test running 5 times\"\n  end\nend\n```\n\nThe `:runner` benchmark will be run 20 times, the default number of runs.  `:benchmark_with_runs` specifies the `:runs` option and will be run only 5 times.\n\n### Running Benchmarks\n\nTo run all benchmarks run:\n\n    $ mix bmark\n\nThis will produce the files\n\n    Project Root\n    +-- results\n    |   +-- example.runner.results\n    |   +-- example.benchmark_with_runs.results\n\nwhich will contain the run times, in miliseconds, for each run of the benchmark.\n\n### Comparing Benchmark Results\n\nIf you have two results files you can compare them by running\n\n    $ mix bmark.cmp results/RunA.results results/RunB.results\n\nand bmark will print out the comparison.  Here's an example of the comparison with explantions for each section:\n\n    RunA:                                 RunB:\n    24274268                              6426990\n    24563751                              6416149\n    24492221                              6507946\n    24516553                              6453309\n    24335224                              6491314\n    24158102                              6405073\n    24357174                              6504260\n    24213098                              6449789\n    24466586                              6532929\n    24289248                              6509800\n\nThe section above contains the raw result data presented side-by-side.  This is the same data your would get by looking at RunA.results and RunB.results.\n\n    24366622.5 -\u003e 6469755.9 (-73.45%) with p \u003c 0.0005\n\nThis line shows the change in mean (average) between the two runs. Next, it shows the percentage change and finally the confidence value.  You can interpret this as saying there is `1 - p`, or a greater than 99.95% confidence that the change in means is statistically significant.  That is, the smaller the value of `p` the more confident you can be in the change in performance.\n\n    t = 391.56626146910503, 18 degrees of freedom\n\nThe final line shows the `t` value and degrees of freedom.  This is the raw statistical data used in Student's t-test to compute the confidence value.\n\n## More Information\n\nBmark development has been described in detail on my blog [Learning Elixir](http://learningelixir.joekain.com/) you can find all the related posts on the [Bmark page](http://learningelixir.joekain.com/bmark-posts/)\n\n## Contributing\n\nSee [Contributing](CONTRIBUTING.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoekain%2Fbmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjoekain%2Fbmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjoekain%2Fbmark/lists"}