{"id":13636200,"url":"https://github.com/BurntSushi/cargo-benchcmp","last_synced_at":"2025-04-19T04:32:21.188Z","repository":{"id":37484191,"uuid":"51722354","full_name":"BurntSushi/cargo-benchcmp","owner":"BurntSushi","description":"A small utility to compare Rust micro-benchmarks.","archived":false,"fork":false,"pushed_at":"2024-08-29T16:45:27.000Z","size":303,"stargazers_count":342,"open_issues_count":8,"forks_count":21,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-10-30T14:04:59.314Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BurntSushi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-15T01:05:58.000Z","updated_at":"2024-09-25T07:38:44.000Z","dependencies_parsed_at":"2024-01-18T04:07:06.879Z","dependency_job_id":"7cbd4d2f-1dde-4fff-8946-623fc3adc46d","html_url":"https://github.com/BurntSushi/cargo-benchcmp","commit_stats":{"total_commits":107,"total_committers":11,"mean_commits":9.727272727272727,"dds":"0.47663551401869164","last_synced_commit":"cc7bd707027e75b7defb75ba1c29697525b177e4"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fcargo-benchcmp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fcargo-benchcmp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fcargo-benchcmp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BurntSushi%2Fcargo-benchcmp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BurntSushi","download_url":"https://codeload.github.com/BurntSushi/cargo-benchcmp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223358302,"owners_count":17132557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T00:00:58.443Z","updated_at":"2024-11-09T05:31:26.139Z","avatar_url":"https://github.com/BurntSushi.png","language":"Rust","funding_links":[],"categories":["Rust","Development tools"],"sub_categories":["Build system"],"readme":"cargo benchcmp\n==============\nA small utility for comparing micro-benchmarks produced by `cargo bench`. The\nutility takes as input two sets of micro-benchmarks (one \"old\" and the other\n\"new\") and shows as output a comparison between each benchmark.\n\n[![Linux build status](https://api.travis-ci.org/BurntSushi/cargo-benchcmp.svg)](https://travis-ci.org/BurntSushi/cargo-benchcmp)\n[![Windows build status](https://ci.appveyor.com/api/projects/status/github/BurntSushi/cargo-benchcmp?svg=true)](https://ci.appveyor.com/project/BurntSushi/cargo-benchcmp)\n[![crates.io page](http://meritbadge.herokuapp.com/cargo-benchcmp)](https://crates.io/crates/cargo-benchcmp)\n\nDual-licensed under MIT or the [UNLICENSE](http://unlicense.org).\n\n### Installation\n\n`cargo benchcmp` can be installed with `cargo install`:\n\n```\n$ cargo install cargo-benchcmp\n```\n\nThe resulting binary should then be in `$HOME/.cargo/bin`.\n\n### Criterion support\n\nThis tool only supports the standard benchmark output emitted by `cargo bench`.\nFor Criterion support, a different tool was developed called\n[critcmp](https://github.com/BurntSushi/critcmp).\n\n### Example output\n\n![Coloured example output on aho-corasick benchmarks](coloured_output_example.png)\n\n### Usage\n\nFirst, run your benchmarks and save them to a file:\n\n```\n$ cargo bench \u003e control\n```\n\nNext, apply the changes you'd like to test out, and then run the benchmarks and\nsave them to a file again.\n\n```\n$ cargo bench \u003e variable\n```\n\nFinally, use `cargo benchcmp` to compare the benchmark results!\n\n```\n$ cargo benchcmp control variable\n name                                bench_output_3.txt ns/iter  bench_output_2.txt ns/iter  diff ns/iter   diff %  speedup\n ac_one_byte                         354 (28248 MB/s)            349 (28653 MB/s)                      -5   -1.41%   x 1.01\n ac_one_prefix_byte_every_match      150,581 (66 MB/s)           112,957 (88 MB/s)                -37,624  -24.99%   x 1.33\n ac_one_prefix_byte_no_match         354 (28248 MB/s)            350 (28571 MB/s)                      -4   -1.13%   x 1.01\n ac_one_prefix_byte_random           20,273 (493 MB/s)           16,096 (621 MB/s)                 -4,177  -20.60%   x 1.26\n ac_ten_bytes                        108,092 (92 MB/s)           58,588 (170 MB/s)                -49,504  -45.80%   x 1.84\n ac_ten_diff_prefix                  108,082 (92 MB/s)           58,601 (170 MB/s)                -49,481  -45.78%   x 1.84\n...\n```\n\nIf you want to compare the same benchmark run in multiple ways, reuse the names\nin different modules. Then your benchmark output will look like:\n\n```\nmodule1::ac_two_one_prefix_byte_random   ...\nmodule2::ac_two_one_prefix_byte_random   ...\n```\n\nYou can then instruct benchcmp to compare the two modules by providing the two\nprefixes, followed by the file containing the output:\n\n```\n$ cargo benchcmp module1:: module2:: benchmark-output\n name                                dense_boxed:: ns/iter  dense:: ns/iter    diff ns/iter   diff %  speedup\n ac_one_byte                         354 (28248 MB/s)       349 (28653 MB/s)             -5   -1.41%   x 1.01\n ac_one_prefix_byte_every_match      150,581 (66 MB/s)      112,957 (88 MB/s)       -37,624  -24.99%   x 1.33\n ac_one_prefix_byte_no_match         354 (28248 MB/s)       350 (28571 MB/s)             -4   -1.13%   x 1.01\n ac_one_prefix_byte_random           20,273 (493 MB/s)      16,096 (621 MB/s)        -4,177  -20.60%   x 1.26\n ac_ten_bytes                        108,092 (92 MB/s)      58,588 (170 MB/s)       -49,504  -45.80%   x 1.84\n ac_ten_diff_prefix                  108,082 (92 MB/s)      58,601 (170 MB/s)       -49,481  -45.78%   x 1.84\n ac_ten_one_prefix_byte_every_match  150,561 (66 MB/s)      112,920 (88 MB/s)       -37,641  -25.00%   x 1.33\n ac_ten_one_prefix_byte_no_match     354 (28248 MB/s)       350 (28571 MB/s)             -4   -1.13%   x 1.01\n ac_ten_one_prefix_byte_random       23,684 (422 MB/s)      19,181 (521 MB/s)        -4,503  -19.01%   x 1.23\n ac_two_bytes                        3,138 (3186 MB/s)      3,125 (3200 MB/s)           -13   -0.41%   x 1.00\n ac_two_diff_prefix                  3,138 (3186 MB/s)      3,124 (3201 MB/s)           -14   -0.45%   x 1.00\n ac_two_one_prefix_byte_every_match  150,571 (66 MB/s)      112,934 (88 MB/s)       -37,637  -25.00%   x 1.33\n ac_two_one_prefix_byte_no_match     354 (28248 MB/s)       350 (28571 MB/s)             -4   -1.13%   x 1.01\n ac_two_one_prefix_byte_random       21,009 (476 MB/s)      16,511 (605 MB/s)        -4,498  -21.41%   x 1.27\n```\n\nThe tool supports basic filtering. For example, it's easy to see only\nimprovements:\n\n```\n$ cargo benchcmp old new --improvements\n name                             full:: ns/iter     full_overlap:: ns/iter  diff ns/iter  diff %  speedup\n ac_one_byte                      367 (27247 MB/s)   367 (27247 MB/s)                   0   0.00%   x 1.00\n ac_two_one_prefix_byte_no_match  371 (26954 MB/s)   368 (27173 MB/s)                  -3  -0.81%   x 1.01\n ac_two_one_prefix_byte_random    11,530 (867 MB/s)  11,514 (868 MB/s)                -16  -0.14%   x 1.00\n```\n\nOr only see regressions:\n\n```\n$ cargo benchcmp old new --regressions\n name                                full:: ns/iter     full_overlap:: ns/iter  diff ns/iter  diff %  speedup\n ac_one_prefix_byte_every_match      27,425 (364 MB/s)  27,972 (357 MB/s)                547   1.99%   x 0.98\n ac_one_prefix_byte_no_match         367 (27247 MB/s)   373 (26809 MB/s)                   6   1.63%   x 0.98\n ac_one_prefix_byte_random           11,076 (902 MB/s)  11,243 (889 MB/s)                167   1.51%   x 0.99\n ac_ten_bytes                        25,474 (392 MB/s)  25,754 (388 MB/s)                280   1.10%   x 0.99\n ac_ten_diff_prefix                  25,466 (392 MB/s)  25,800 (387 MB/s)                334   1.31%   x 0.99\n ac_ten_one_prefix_byte_every_match  27,424 (364 MB/s)  28,046 (356 MB/s)                622   2.27%   x 0.98\n ac_ten_one_prefix_byte_no_match     367 (27247 MB/s)   369 (27100 MB/s)                   2   0.54%   x 0.99\n ac_ten_one_prefix_byte_random       13,661 (732 MB/s)  13,742 (727 MB/s)                 81   0.59%   x 0.99\n ac_two_bytes                        3,141 (3183 MB/s)  3,164 (3160 MB/s)                 23   0.73%   x 0.99\n ac_two_diff_prefix                  3,141 (3183 MB/s)  3,174 (3150 MB/s)                 33   1.05%   x 0.99\n ac_two_one_prefix_byte_every_match  27,638 (361 MB/s)  27,953 (357 MB/s)                315   1.14%   x 0.99\n\n```\n\nMany times, the difference in micro-benchmarks is just noise, so you can filter\nby percent difference:\n\n```\n$ cargo benchcmp old new --regressions --threshold 2\n name                                full:: ns/iter     full_overlap:: ns/iter  diff ns/iter  diff %  speedup\n ac_ten_one_prefix_byte_every_match  27,424 (364 MB/s)  28,046 (356 MB/s)                622   2.27%   x 0.98\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBurntSushi%2Fcargo-benchcmp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBurntSushi%2Fcargo-benchcmp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBurntSushi%2Fcargo-benchcmp/lists"}