{"id":13484425,"url":"https://github.com/evanphx/benchmark-ips","last_synced_at":"2025-05-14T22:04:44.572Z","repository":{"id":2813297,"uuid":"3814699","full_name":"evanphx/benchmark-ips","owner":"evanphx","description":"Provides iteration per second benchmarking for Ruby","archived":false,"fork":false,"pushed_at":"2025-04-14T02:46:32.000Z","size":298,"stargazers_count":1748,"open_issues_count":5,"forks_count":98,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-05-07T21:13:24.549Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/evanphx.png","metadata":{"files":{"readme":"README.md","changelog":"History.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2012-03-24T02:58:55.000Z","updated_at":"2025-05-05T02:31:46.000Z","dependencies_parsed_at":"2023-12-12T19:56:30.451Z","dependency_job_id":"54da2683-101b-4f6a-808a-7bd460b5bc3e","html_url":"https://github.com/evanphx/benchmark-ips","commit_stats":{"total_commits":210,"total_committers":45,"mean_commits":4.666666666666667,"dds":0.8047619047619048,"last_synced_commit":"6c0a6bfc823f1c3f8fe3a8986cc395e5e13ae253"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evanphx%2Fbenchmark-ips","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evanphx%2Fbenchmark-ips/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evanphx%2Fbenchmark-ips/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/evanphx%2Fbenchmark-ips/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/evanphx","download_url":"https://codeload.github.com/evanphx/benchmark-ips/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253416098,"owners_count":21904983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:24.228Z","updated_at":"2025-05-14T22:04:44.500Z","avatar_url":"https://github.com/evanphx.png","language":"Ruby","readme":"# benchmark-ips\n\n* rdoc :: http://rubydoc.info/gems/benchmark-ips\n* home :: https://github.com/evanphx/benchmark-ips\n\n[![Gem Version](https://badge.fury.io/rb/benchmark-ips.svg)](http://badge.fury.io/rb/benchmark-ips)\n[![CI](https://github.com/evanphx/benchmark-ips/actions/workflows/ci.yml/badge.svg)](https://github.com/evanphx/benchmark-ips/actions/workflows/ci.yml)\n[![Inline docs](http://inch-ci.org/github/evanphx/benchmark-ips.svg)](http://inch-ci.org/github/evanphx/benchmark-ips)\n\n* https://github.com/evanphx/benchmark-ips\n\n## DESCRIPTION:\n\nAn iterations per second enhancement to Benchmark.\n\n## FEATURES/PROBLEMS:\n\n * benchmark/ips - benchmarks a blocks iterations/second. For short snippits\n   of code, ips automatically figures out how many times to run the code\n   to get interesting data. No more guessing at random iteration counts!\n\n## SYNOPSIS:\n\n```ruby\nrequire 'benchmark/ips'\n\nBenchmark.ips do |x|\n  # Configure the number of seconds used during\n  # the warmup phase (default 2) and calculation phase (default 5)\n  x.config(warmup: 2, time: 5)\n\n  # Typical mode, runs the block as many times as it can\n  x.report(\"addition\") { 1 + 2 }\n\n  # To reduce overhead, the number of iterations is passed in\n  # and the block must run the code the specific number of times.\n  # Used for when the workload is very small and any overhead\n  # introduces incorrectable errors.\n  x.report(\"addition2\") do |times|\n    i = 0\n    while i \u003c times\n      i += 1\n\n      1 + 2\n    end\n  end\n\n  # To reduce overhead even more, grafts the code given into\n  # the loop that performs the iterations internally to reduce\n  # overhead. Typically not needed, use the |times| form instead.\n  x.report(\"addition3\", \"1 + 2\")\n\n  # Really long labels should be formatted correctly\n  x.report(\"addition-test-long-label\") { 1 + 2 }\n\n  # Compare the iterations per second of the various reports!\n  x.compare!\nend\n```\n\nThis will generate the following report:\n\n```\nWarming up --------------------------------------\n            addition     3.572M i/100ms\n           addition2     3.672M i/100ms\n           addition3     3.677M i/100ms\naddition-test-long-label\n                         3.511M i/100ms\nCalculating -------------------------------------\n            addition     36.209M (± 2.8%) i/s   (27.62 ns/i) -    182.253M in   5.037433s\n           addition2     36.552M (± 7.8%) i/s   (27.36 ns/i) -    183.541M in   5.069987s\n           addition3     36.639M (± 4.8%) i/s   (27.29 ns/i) -    182.994M in   5.009234s\naddition-test-long-label\n                         36.164M (± 5.8%) i/s   (27.65 ns/i) -    181.312M in   5.038364s\n\nComparison:\n           addition2: 36558904.5 i/s\n           addition3: 36359284.0 i/s - same-ish: difference falls within error\naddition-test-long-label: 36135428.8 i/s - same-ish: difference falls within error\n            addition: 34666931.3 i/s - same-ish: difference falls within error\n```\n\nBenchmark/ips will report the number of iterations per second for a given block\nof code. When analyzing the results, notice the percent of [standard\ndeviation](http://en.wikipedia.org/wiki/Standard\\_deviation) which tells us how\nspread out our measurements are from the average. A high standard deviation\ncould indicate the results having too much variability.\n\nOne benefit to using this method is benchmark-ips automatically determines the\ndata points for testing our code, so we can focus on the results instead of\nguessing iteration counts as we do with the traditional Benchmark library.\n\nYou can also use `ips_quick` to save a few lines of code:\n\n```ruby\nBenchmark.ips_quick(:upcase, :downcase, on: \"hello\") # runs a suite comparing \"hello\".upcase and \"hello\".downcase\n\ndef first; MyJob.perform(1); end\ndef second; MyJobOptimized.perform(1); end\nBenchmark.ips_quick(:first, :second) # compares :first and :second\n```\n\nThis adds a very small amount of overhead, which may be significant (i.e. ips_quick will understate the difference) if you're microbenchmarking things that can do over 1 million iterations per second. In that case, you're better off using the full format.\n\n### Custom Suite\n\nPass a custom suite to disable garbage collection during benchmark:\n\n```ruby\nrequire 'benchmark/ips'\n\n# Enable and start GC before each job run. Disable GC afterwards.\n#\n# Inspired by https://www.omniref.com/ruby/2.2.1/symbols/Benchmark/bm?#annotation=4095926\u0026line=182\nclass GCSuite\n  def warming(*)\n    run_gc\n  end\n\n  def running(*)\n    run_gc\n  end\n\n  def warmup_stats(*)\n  end\n\n  def add_report(*)\n  end\n\n  private\n\n  def run_gc\n    GC.enable\n    GC.start\n    GC.disable\n  end\nend\n\nsuite = GCSuite.new\n\nBenchmark.ips do |x|\n  x.config(:suite =\u003e suite)\n  x.report(\"job1\") { ... }\n  x.report(\"job2\") { ... }\nend\n```\n\n### Independent benchmarking\n\nIf you are comparing multiple implementations of a piece of code you may want\nto benchmark them in separate invocations of Ruby so that the measurements\nare independent of each other. You can do this with the `hold!` command.\n\n```ruby\nBenchmark.ips do |x|\n\n  # Hold results between multiple invocations of Ruby\n  x.hold! 'filename'\n\nend\n```\n\nThis will run only one benchmarks each time you run the command, storing\nresults in the specified file. The file is deleted when all results have been\ngathered and the report is shown.\n\nAlternatively, if you prefer a different approach, the `save!` command is\navailable. Examples for [hold!](examples/hold.rb) and [save!](examples/save.rb) are available in\nthe `examples/` directory.\n\n\n### Multiple iterations\n\nIn some cases you may want to run multiple iterations of the warmup and\ncalculation stages and take only the last result for comparison. This is useful\nif you are benchmarking with an implementation of Ruby that optimizes using\ntracing or on-stack-replacement, because to those implementations the\ncalculation phase may appear as new, unoptimized code.\n\nYou can do this with the `iterations` option, which by default is `1`. The\ntotal time spent will then be `iterations * warmup + iterations * time` seconds.\n\n```ruby\nBenchmark.ips do |x|\n\n  x.config(:iterations =\u003e 3)\n\n    # or\n\n  x.iterations = 3\n\nend\n```\n\n### Online sharing\n\nIf you want to quickly share your benchmark result with others, run you benchmark\nwith `SHARE=1` argument. For example: `SHARE=1 ruby my_benchmark.rb`.\n\nResult will be sent to [benchmark.fyi](https://ips.fastruby.io/) and benchmark-ips\nwill display the link to share the benchmark's result.\n\nIf you want to run your own instance of [benchmark.fyi](https://github.com/evanphx/benchmark.fyi)\nand share it to that instance, you can do this: `SHARE_URL=https://ips.example.com ruby my_benchmark.rb`\n\n### Advanced Statistics\n\nBy default, the margin of error shown is plus-minus one standard deviation. If\na more advanced statistical test is wanted, a bootstrap confidence interval\ncan be calculated instead. A bootstrap confidence interval has the advantages of\narguably being more mathematically sound for this application than a standard\ndeviation, it additionally produces an error for relative slowdowns, which the\nstandard deviation does not, and it is arguably more intuitive and actionable.\n\nWhen a bootstrap confidence interval is used, a median of the interval is used\nrather than the mean of the samples, which is what you get with the default\nstandard deviation.\n\nThe bootstrap confidence interval used is the one described by Tomas Kalibera.\nNote that for this technique to be valid your benchmark should have reached a\nnon-periodic steady state with statistically independent samples (it should\nhave warmed up) by the time measurements start.\n\nUsing a bootstrap confidence internal requires that the 'kalibera' gem is\ninstalled separately. This gem is not a formal dependency, as by default it is\nnot needed.\n\n```\ngem install kalibera\n```\n\n```ruby\nBenchmark.ips do |x|\n\n  # The default is :stats =\u003e :sd, which doesn't have a configurable confidence\n  x.config(:stats =\u003e :bootstrap, :confidence =\u003e 95)\n\n    # or\n\n  x.stats = :bootstrap\n  x.confidence = 95\n\n  # confidence is 95% by default, so it can be omitted\n\nend\n```\n\n### Output as JSON\n\nYou can generate output in JSON. If you want to write JSON to a file, pass filename to `json!` method:\n\n```ruby\nBenchmark.ips do |x|\n  x.report(\"some report\") {  }\n  x.json! 'filename.json'\nend\n```\n\nIf you want to write JSON to STDOUT, pass `STDOUT` to `json!` method and set `quiet = true` before `json!`:\n\n```ruby\nBenchmark.ips do |x|\n  x.report(\"some report\") {  }\n  x.quiet = true\n  x.json! STDOUT\nend\n```\n\nThis is useful when the output from `benchmark-ips` becomes an input of other tools via stdin.\n\n## REQUIREMENTS:\n\n* None!\n\n## INSTALL:\n\n    $ gem install benchmark-ips\n\n## DEVELOPERS:\n\nAfter checking out the source, run:\n\n    $ rake newb\n\nThis task will install any missing dependencies, run the tests/specs,\nand generate the RDoc.\n\n","funding_links":[],"categories":["Ruby","1. language","Profiler and Optimization","Profiler","Performance tools"],"sub_categories":["1.1 ruby"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevanphx%2Fbenchmark-ips","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fevanphx%2Fbenchmark-ips","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fevanphx%2Fbenchmark-ips/lists"}