{"id":20100604,"url":"https://github.com/seriyps/rebar3_bench","last_synced_at":"2025-04-07T10:28:28.055Z","repository":{"id":35103772,"uuid":"207182764","full_name":"seriyps/rebar3_bench","owner":"seriyps","description":"Microbenchmark plugin for rebar3","archived":false,"fork":false,"pushed_at":"2024-10-14T18:37:06.000Z","size":69,"stargazers_count":19,"open_issues_count":5,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T09:05:48.464Z","etag":null,"topics":["benchmark","criterion","erlang","microbenchmark","rebar3-plugin"],"latest_commit_sha":null,"homepage":null,"language":"Erlang","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/seriyps.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-08T22:42:59.000Z","updated_at":"2024-10-14T18:33:35.000Z","dependencies_parsed_at":"2024-12-17T10:04:44.351Z","dependency_job_id":"f7caed78-f100-44c2-ba4b-87ffc80dbfb0","html_url":"https://github.com/seriyps/rebar3_bench","commit_stats":{"total_commits":14,"total_committers":3,"mean_commits":4.666666666666667,"dds":0.2142857142857143,"last_synced_commit":"bbb246abbd85bb84a2e4bd79183bc07326ee4a38"},"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seriyps%2Frebar3_bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seriyps%2Frebar3_bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seriyps%2Frebar3_bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/seriyps%2Frebar3_bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/seriyps","download_url":"https://codeload.github.com/seriyps/rebar3_bench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247634894,"owners_count":20970627,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","criterion","erlang","microbenchmark","rebar3-plugin"],"created_at":"2024-11-13T17:16:27.502Z","updated_at":"2025-04-07T10:28:28.026Z","avatar_url":"https://github.com/seriyps.png","language":"Erlang","funding_links":[],"categories":[],"sub_categories":[],"readme":"rebar3 bench\n============\n\nRebar3 microbenchmark plugin\n\nWas inspired by Haskell [criterion](https://hackage.haskell.org/package/criterion) and\nRust's [criterion.rs](https://crates.io/crates/criterion).\nParts of implementation were borrowed from [rebar3 proper](https://hex.pm/packages/rebar3_proper) plugin.\nIt relies on [eministat](https://hex.pm/packages/eministat) for statistical calculations.\n\nUse\n---\n\nAdd the plugin to your rebar config:\n\n```\n{project_plugins, [rebar3_bench]}.\n```\n\nAdd `bench_*` modules to `\"test\"` directory of your application:\n\n```\nmy_app/\n├── README.md\n├── rebar.config\n├── src\n└── test\n    ├── bench_codec.erl\n    ├── bench_kv.erl\n    └── bench_calc.erl\n```\n\nEach module should have one or more exported pairs of functions:\n\n* `bench_\u003cname\u003e/1` function which prepares the input and state data for the benchmark (called before\n  and after the benchmark)\n* `\u003cname\u003e/2` function that is the body of the benchmark - called hundreds of times\n\nNOTE: in the older versions of `rebar3_bench` the naming of the callbacks was opposite!\nSee `callback_style` option.\n\n```\n-export([bench_reverse/1,\n         reverse/2,\n         bench_sort/1,\n         sort/2]).\n\nbench_reverse({input, _}) -\u003e\n    lists:seq(1, 1000).\n\nreverse(List, _) -\u003e\n    lists:reverse(List).\n\n\nbench_sort({input, _}) -\u003e\n    lists:seq(1, 1000).\n\nsort(List, _) -\u003e\n    lists:sort(List).\n\n\nbench_server(init) -\u003e\n    % init is called only once in the same process where benchmark will be running\n    % at the very beginning of benchmark.\n    % Value returned from this callback will be passed to server({input, State}),\n    % bench_server(_, State) and server({stop, State})\n    my_server:start();\nbench_server({input, _Server}) -\u003e\n    % This callback is called after `init` to generate benchmark input data.\n    % Returned value will be passed to bench_server(Input, _)\n    binary:copy(\u003c\u003c1\u003e\u003e, 1024);\nbench_server({stop, Server}) -\u003e\n    % Called only once at the very end of benchmark\n    my_server:stop(Server).\n\nserver(BinaryInput, Server) -\u003e\n    my_server:send(BinaryInput, Server),\n    BinaryInput = my_server:recv(Server).\n```\n\nEach benchmark is executed in a separate process with `{priority, high}, {min_heap_size, 5mb}`.\nGarbage-collection is forced before time measurement is started, but if your benchmark\nfunction produces a lot of heap data and benchmark duration (`-t` option) is high enough, it's\nstill possible that garbage collection will be triggered during benchmark execution.\n\nIt's importand that input and benchmark are as deterministic as possible. Try to not depend on\nuse of random generators, time or other side effects.\n\nThen just call your plugin directly in an existing application (it will auto-discover your benchmarks):\n\n```\n$ rebar3 bench --callback-style=new\n===\u003e Testing bench_kv:bench_maps()\n===\u003e Stats for wall_time\nMin:              133.00ns\n25 percentile:    133.00ns\nMedian:           134.00ns\n75 percentile:    135.00ns\nMax:              174.00ns\nOutliers:       Lo: 0; Hi: 8; Sum: 8\nOutlier variance:      0.316545 (moderate)\n\u003e Bootstrapped\nMean:             135.00ns\nStd deviation:      5.00ns\n```\n\nPlease, refer to [eministat README](https://github.com/jlouis/eministat#description-of-the-output)\nto find out what does the output mean.\n\nThis will collect 100 samples, dump them to special directory under `_build` and calculates\nstatistics over them.\n\nIf you make your optimizations and run the same command again, it will\nagain collect 100 samples using your new code, calculates statistics, comparing\nresults with previous dump and will try to statistically prove that there exists\nsignificant difference between results of two runs - if there is an improvement or\nregression:\n\n```\n$ rebar3 bench --callback-style=new\n===\u003e Testing bench_kv:bench_maps()\n===\u003e Stats for wall_time\nMin:                133.00ns (+    0.00ns /  0.3%)\n25 percentile:      134.00ns (+    1.00ns /  0.4%)\nMedian:             134.00ns (-    0.00ns /  0.1%)\n75 percentile:      135.00ns (-    7.00ns /  4.9%)\nMax:                146.00ns (-    0.00ns /  0.1%)\nOutliers:         Lo: 0; Hi: 5; Sum: 5\nOutlier variance: 9.90000e-3 (unaffected)\n\u003e Bootstrapped\nMean:               135.00ns (-    2.00ns /  1.6%)\nStd deviation:        2.00ns\n\u003e Relative\nDifference at 95.0 confidence\n    -2.00ns ±     1.00ns\n    -1.60%  ±  0.77%\n (Student's t, pooled s = 3.21558)\n```\n\nThe most important field to pay attention is `Mean`: `135.00ns (-    2.00ns /  1.6%)` means\nthat the runtime of the single call to benchmarked function decreased by 2ns (1.6%), which\nis a very minor improvement.\n\nHowever, in the section below `\u003eRelative` the statistical proof says, that this improvement is\nnot caused by random fluctuations, but is quite stable - \"proven\".\n\nAnother important data point is `Outlier variance`. If the value is more than ~0.5, it means\nthat the runtime of benchmarked function varies a lot between calls. So you need to either\nsimplify this function, remove side-effects from it or increase the benchmark runtime to\nsmoothen the influence of side-effects.\n\n### Example workflow\n\nYou can use `--dump-baseline` and `--baseline` to manage benchmark samples dumps.\n\n```\ngit checkout master\n# run benchmark and store results in \"master\" file\nrebar3 bench --save-baseline master\ngit checkout optimizations\n# run benchmark and compare results with data from \"master\" file\nrebar3 bench --baseline master\n\n# Do some optimizations\n\n# run again and compare with \"master\" data to see if optimizations do work\nrebar3 bench --baseline master\n```\n\n### Cover mode\n\nIt might be interesting to see which lines of code your benchmark actually covers and what are\nthe most \"hot\" code paths. You can run your benchmark with \"cover\" enabled:\n`rebar3 bench --cover`. It will run your benchmarks and will write coverage data to rebar3\nstandard destination, you can use `rebar3 cover` to see the report.\nWe don't do any measurements calculate statistics or write baseline files in `cover` mode,\nbecause cover-compiled code is much slower.\n\n### Available options\n\n```\n$ rebar3 help bench\nRun microbenchmarks from callback modules\nUsage: rebar3 bench [-d [\u003cdir\u003e]] [-m \u003cmodule\u003e] [-b \u003cbenches\u003e]\n                    [-t [\u003cduration\u003e]] [-n [\u003csamples\u003e]]\n                    [--confidence [\u003cconfidence\u003e]] [-c [\u003ccover\u003e]]\n                    [-p [\u003cparameter\u003e]]\n                    [--save-baseline [\u003csave_baseline\u003e]]\n                    [--baseline [\u003cbaseline\u003e]]\n                    [--callback-style [\u003ccallback_style\u003e]]\n\n  -d, --dir          directory where the benchmark tests are located \n                     [default: test]\n  -m, --module       name of one or more modules to run (comma-separated)\n  -b, --bench        name of benchmark to run within a specified module \n                     (comma-separated)\n  -t, --duration     duration of single benchmark, in seconds [default: 10]\n  -n, --num-samples  number of samples to collect and analyze [default: \n                     100]\n  --confidence       confidence level: 80, 90, 95, 98, 99 [default: 95]\n  -c, --cover        run benchmarks in coverage mode (no measurements are \n                     made), generate cover data [default: false]\n  -p, --parameter    which parameter to measure: wall_time, memory, \n                     reductions [default: wall_time]\n  --save-baseline    save benchmark data to file with this name [default: \n                     _tip]\n  --baseline         use data stored in file as baseline [default: _tip]\n  --callback-style   naming of benchmark and benchmark options functions\n                     (new | legacy) [default: legacy]\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseriyps%2Frebar3_bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fseriyps%2Frebar3_bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fseriyps%2Frebar3_bench/lists"}