{"id":13521133,"url":"https://github.com/piotrmurach/rspec-benchmark","last_synced_at":"2025-04-13T02:01:11.426Z","repository":{"id":43061451,"uuid":"49822782","full_name":"piotrmurach/rspec-benchmark","owner":"piotrmurach","description":"Performance testing matchers for RSpec","archived":false,"fork":false,"pushed_at":"2024-03-16T22:22:56.000Z","size":211,"stargazers_count":602,"open_issues_count":3,"forks_count":24,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-10-29T15:23:33.772Z","etag":null,"topics":["benchmark","measurements","performance-testing","rspec","rspec-matchers","testing"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/piotrmurach.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"piotrmurach"}},"created_at":"2016-01-17T15:24:36.000Z","updated_at":"2024-10-07T00:45:22.000Z","dependencies_parsed_at":"2024-06-12T16:29:42.010Z","dependency_job_id":null,"html_url":"https://github.com/piotrmurach/rspec-benchmark","commit_stats":{"total_commits":271,"total_committers":8,"mean_commits":33.875,"dds":"0.059040590405904037","last_synced_commit":"b6241821e3066a9b9d3efa090c1cb0ac0657bda3"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotrmurach%2Frspec-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotrmurach%2Frspec-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotrmurach%2Frspec-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/piotrmurach%2Frspec-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/piotrmurach","download_url":"https://codeload.github.com/piotrmurach/rspec-benchmark/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248654045,"owners_count":21140235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","measurements","performance-testing","rspec","rspec-matchers","testing"],"created_at":"2024-08-01T06:00:29.102Z","updated_at":"2025-04-13T02:01:11.396Z","avatar_url":"https://github.com/piotrmurach.png","language":"Ruby","funding_links":["https://github.com/sponsors/piotrmurach"],"categories":["Ruby","Matchers"],"sub_categories":[],"readme":"# RSpec::Benchmark\n\n[![Gem Version](https://badge.fury.io/rb/rspec-benchmark.svg)][gem]\n[![Actions CI](https://github.com/piotrmurach/rspec-benchmark/workflows/CI/badge.svg?branch=master)][gh_actions_ci]\n[![Build status](https://ci.appveyor.com/api/projects/status/nxq3dr8xkafmgiv0?svg=true)][appveyor]\n[![Code Climate](https://codeclimate.com/github/piotrmurach/rspec-benchmark/badges/gpa.svg)][codeclimate]\n[![Coverage Status](https://coveralls.io/repos/github/piotrmurach/rspec-benchmark/badge.svg)][coverage]\n[![Inline docs](https://inch-ci.org/github/piotrmurach/rspec-benchmark.svg?branch=master)][inchpages]\n\n[gem]: https://badge.fury.io/rb/rspec-benchmark\n[gh_actions_ci]: https://github.com/piotrmurach/rspec-benchmark/actions?query=workflow%3ACI\n[appveyor]: https://ci.appveyor.com/project/piotrmurach/rspec-benchmark\n[codeclimate]: https://codeclimate.com/github/piotrmurach/rspec-benchmark\n[coverage]: https://coveralls.io/github/piotrmurach/rspec-benchmark\n[inchpages]: https://inch-ci.org/github/piotrmurach/rspec-benchmark\n\n\u003e Performance testing matchers for RSpec to set expectations on speed, resources usage and scalability.\n\n**RSpec::Benchmark** is powered by:\n\n* [benchmark-perf](https://github.com/piotrmurach/benchmark-perf) for measuring execution time and iterations per second.\n* [benchmark-trend](https://github.com/piotrmurach/benchmark-trend) for estimating computation complexity.\n* [benchmark-malloc](https://github.com/piotrmurach/benchmark-malloc) for measuring object and memory allocations.\n\n## Why?\n\nIntegration and unit tests ensure that changing code maintains expected functionality. What is not guaranteed is the code changes impact on library performance. It is easy to refactor your way out of fast to slow code.\n\nIf you are new to performance testing you may find [Caveats](#5-caveats) section helpful.\n\n## Contents\n\n* [1. Usage](#1-usage)\n  * [1.1 Timing](#11-timing)\n  * [1.2 Iterations ](#12-iterations)\n  * [1.3 Comparison ](#13-comparison)\n  * [1.4 Complexity](#14-complexity)\n  * [1.5 Allocation](#15-allocation)\n* [2. Compounding](#2-compounding)\n* [3. Configuration](#3-configuration)\n  * [3.1 :disable_gc](#31-disable_gc)\n  * [3.2 :run_in_subprocess](#32-run_in_subprocess)\n  * [3.3 :samples](#33-samples)\n  * [3.4 :format](#34-format)\n* [4. Filtering](#4-filtering)\n* [5. Caveats](#5-caveats)\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'rspec-benchmark'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install rspec-benchmark\n\n## 1. Usage\n\nFor matchers to be available globally, in `spec_helper.rb` do:\n\n```ruby\nrequire 'rspec-benchmark'\n\nRSpec.configure do |config|\n  config.include RSpec::Benchmark::Matchers\nend\n```\n\nThis will add the following matchers:\n\n* `perform_under` to see how fast your code runs\n* `perform_at_least` to see how many iteration per second your code can do\n* `perform_(faster|slower)_than` to compare implementations\n* `perform_(constant|linear|logarithmic|power|exponential)` to see how your code scales with time\n* `perform_allocation` to limit object and memory allocations\n\nThese will help you express expected performance benchmark for an evaluated code.\n\nAlternatively, you can add matchers for particular example:\n\n```ruby\nRSpec.describe \"Performance testing\" do\n  include RSpec::Benchmark::Matchers\nend\n```\n\nThen you're good to start setting performance expectations:\n\n```ruby\nexpect {\n  ...\n}.to perform_under(6).ms\n```\n\n### 1.1 Timing\n\nThe `perform_under` matcher answers the question of how long does it take to perform a given block of code on average. The measurements are taken executing the block of code in a child process for accurate CPU times.\n\n```ruby\nexpect { ... }.to perform_under(0.01).sec\n```\n\nAll measurements are assumed to be expressed as seconds. However, you can also provide time in `ms`, `us` and `ns`. The equivalent example in `ms` would be:\n\n```ruby\nexpect { ... }.to perform_under(10).ms\nexpect { ... }.to perform_under(10000).us\n```\n\nBy default the above code will be sampled only once but you can change this by using the `sample` matcher like so:\n\n```ruby\nexpect { ... }.to perform_under(0.01).sample(10) # repeats measurements 10 times\n```\n\nFor extra expressiveness you can use `times`:\n\n```ruby\nexpect { ... }.to perform_under(0.01).sample(10).times\n```\n\nYou can also use `warmup` matcher that can run your code before the actual samples are taken to reduce erratic execution times.\n\nFor example, you can execute code twice before you take 10 actual measurements:\n\n```ruby\nexpect { ... }.to perform_under(0.01).sec.warmup(2).times.sample(10).times\n```\n\n### 1.2 Iterations\n\nThe `perform_at_least` matcher allows you to establish performance benchmark of how many iterations per second a given block of code should perform. For example, to expect a given code to perform at least 10K iterations per second do:\n\n```ruby\nexpect { ... }.to perform_at_least(10000).ips\n```\n\nThe `ips` part is optional but its usage clarifies the intent.\n\nThe performance timing of this matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds.\n\nBy default `within` matcher is set to `0.2` second and `warmup` matcher to `0.1` respectively. To change how long measurements are taken, for example, to double the time amount do:\n\n```ruby\nexpect { ... }.to perform_at_least(10000).within(0.4).warmup(0.2).ips\n```\n\nThe higher values for `within` and `warmup` the more accurate average readings and more stable tests at the cost of longer test suite overall runtime.\n\n### 1.3 Comparison\n\nThe `perform_faster_than` and `perform_slower_than` matchers allow you to test performance of your code compared with other. For example:\n\n```ruby\nexpect { ... }.to perform_faster_than { ... }\nexpect { ... }.to perform_slower_than { ... }\n```\n\nAnd if you want to compare how much faster or slower your code is do:\n\n```ruby\nexpect { ... }.to perform_faster_than { ... }.once\nexpect { ... }.to perform_faster_than { ... }.twice\nexpect { ... }.to perform_faster_than { ... }.exactly(5).times\nexpect { ... }.to perform_faster_than { ... }.at_least(5).times\nexpect { ... }.to perform_faster_than { ... }.at_most(5).times\n\nexpect { ... }.to perform_slower_than { ... }.once\nexpect { ... }.to perform_slower_than { ... }.twice\nexpect { ... }.to perform_slower_than { ... }.at_least(5).times\nexpect { ... }.to perform_slower_than { ... }.at_most(5).times\nexpect { ... }.to perform_slower_than { ... }.exactly(5).times\n```\n\nThe `times` part is also optional.\n\nThe performance timing of each matcher can be tweaked using the `within` and `warmup` matchers. These are expressed as seconds. By default `within` matcher is set to `0.2` and `warmup` matcher to `0.1` second respectively. To change these matchers values do:\n\n```ruby\nexpect { ... }.to perform_faster_than { ... }.within(0.4).warmup(0.2)\n```\n\nThe higher values for `within` and `warmup` the more accurate average readings and more stable tests at the cost of longer test suite overall runtime.\n\n### 1.4 Complexity\n\nThe `perform_constant`, `perform_logarithmic`, `perform_linear`, `perform_power` and `perform_exponential` matchers are useful for estimating the asymptotic behaviour of a given block of code. The most basic way to use the expectations to test how your code scales is to use the matchers:\n\n```ruby\nexpect { ... }.to perform_constant\nexpect { ... }.to perform_logarithmic/perform_log\nexpect { ... }.to perform_linear\nexpect { ... }.to perform_power\nexpect { ... }.to perform_exponential/perform_exp\n```\n\nTo test performance in terms of computation complexity you can follow the algorithm:\n\n1. Choose a method to profile.\n2. Choose workloads for the method.\n3. Describe workloads with input features.\n4. Assert the performance in terms of Big-O notation.\n\nOften, before expectation can be set you need to setup some workloads. To create a range of inputs use the `bench_range` helper method.\n\nFor example, to create a power range of inputs from `8` to `100_000` do:\n\n```ruby\nsizes = bench_range(8, 100_000) # =\u003e [8, 64, 512, 4096, 32768, 100000]\n```\n\nThen you can use the sizes to create test data, for example to check Ruby's `max` performance create array of number arrays.\n\n```ruby\nnumber_arrays = sizes.map { |n| Array.new(n) { rand(n) } }\n```\n\nUsing `in_range` matcher you can inform the expectation about the inputs. Each range value together with its index will be yielded as arguments to the evaluated block.\n\nYou can either specify the range limits:\n\n```ruby\nexpect { |n, i|\n  number_arrays[i].max\n}.to perform_linear.in_range(8, 100_000)\n```\n\nOr use previously generated `sizes` array:\n\n```ruby\nexpect { |n, i|\n  number_arrays[i].max\n}.to perform_linear.in_range(sizes)\n```\n\nThis example will generate and yield input `n` and index `i` pairs `[8, 0]`, `[64, 1]`, `[512, 2]`, `[4K, 3]`, `[32K, 4]` and `[100K, 5]` respectively.\n\nBy default the range will be generated using ratio of `8`. You can change this using `ratio` matcher:\n\n```ruby\nexpect { |n, i|\n  number_arrays[i].max\n}.to perform_linear.in_range(8, 100_000).ratio(2)\n```\n\nThe performance measurements for a code block are taken only once per range input. You can increase the stability of your performance test by using the `sample` matcher. For example, to repeat measurements 100 times for each range input do:\n\n```ruby\nexpect { |n, i|\n  number_arrays[i].max\n}.to perform_linear.in_range(8, 100_000).ratio(2).sample(100).times\n```\n\nThe overall quality of the performance trend is assessed using a threshold value where `0` means a poor fit and `1` a perfect fit. By default this value is configured to `0.9` as a 'good enough' threshold. To change this use `threshold` matcher:\n\n```ruby\nexpect { |n, i|\n  number_arrays[i].max\n}.to perform_linear.in_range(8, 100_000).threshold(0.95)\n```\n\n### 1.5 Allocation\n\nThe `perform_allocation` matcher checks how much memory or objects have been allocated during a piece of Ruby code execution.\n\nBy default the matcher verifies the number of object allocations. The specified number serves as the _upper limit_ of allocations, so your tests won't become brittle as different Ruby versions change internally how many objects are allocated for some operations.\n\nNote that you can also check for memory allocation using the `bytes` matcher.\n\nTo check number of objects allocated do:\n\n```ruby\nexpect {\n  [\"foo\", \"bar\", \"baz\"].sort[1]\n}.to perform_allocation(3)\n```\n\nYou can also be more granular with your object allocations and specify which object types you're interested in:\n\n```ruby\nexpect {\n  _a = [Object.new]\n  _b = {Object.new =\u003e 'foo'}\n}.to perform_allocation({Array =\u003e 1, Object =\u003e 2}).objects\n```\n\nAnd you can also check how many objects are left when expectation finishes to ensure that `GC` is able to collect them.\n\n```ruby\nexpect {\n  [\"foo\", \"bar\", \"baz\"].sort[1]\n}.to perform_allocation(3).and_retain(3)\n```\n\nYou can also set expectations on the memory size. In this case the memory size will serve as upper limit for the expectation:\n\n```ruby\nexpect {\n  _a = [Object.new]\n  _b = {Object.new =\u003e 'foo'}\n}.to perform_allocation({Array =\u003e 40, Hash =\u003e 384, Object =\u003e 80}).bytes\n```\n\n## 2. Compounding\n\nAll the matchers can be used in compound expressions via `and/or`. For example, if you wish to check if a computation performs under certain time boundary and iterates at least a given number do:\n\n```ruby\nexpect {\n  ...\n}.to perform_under(6).ms and perform_at_least(10000).ips\n```\n\n## 3. Configuration\n\nBy default the following configuration is used:\n\n```ruby\nRSpec::Benchmark.configure do |config|\n  config.run_in_subprocess = false\n  config.disable_gc = false\nend\n```\n\n### 3.1. `:disable_gc`\n\nBy default all tests are run with `GC` enabled. We want to measure real performance or Ruby code. However, disabling `GC` may lead to much quicker test execution. You can change this setting:\n\n```ruby\nRSpec::Benchmark.configure do |config|\n  config.disable_gc = true\nend\n```\n\n### 3.2 `:run_in_subprocess`\n\nThe `perform_under` matcher can run all the measurements in the subprocess. This will increase isolation from other processes activity. However, the `rspec-rails` gem runs all tests in transactions. Unfortunately, when running tests in child process, database connections are used from connection pool and no data can be accessed. This is only a problem when running specs in Rails. Any other Ruby project can run specs using subprocesses. To enable this behaviour do:\n\n```ruby\nRSpec::Benchmark.configure do |config|\n  config.run_in_subprocess = true\nend\n```\n\n### 3.3 `:samples`\n\nThe `perform_under` and computational complexity matchers allow to specify how many times to repeat measurements. You configure it globally for all matchers using the `:samples` option which defaults to `1`:\n\n```ruby\nRSpec::Benchmark.configure do |config|\n  config.samples = 10\nend\n```\n\n### 3.4 `:format`\n\nThe `perform_at_least` matcher uses the `:format` option to format the number of iterations when a failure message gets displayed. By default, the `:human` values is used to make numbers more readable. For example, the `12300 i/s` gets turned into `12.3k i/s`. If you rather have an exact numbers presented do:\n\n```ruby\nRSpec::Benchmark.configure do |config|\n  config.format = :raw\nend\n```\n\n## 4. Filtering\n\nUsually performance tests are best left for CI or occasional runs that do not affect TDD/BDD cycle.\n\nTo achieve isolation you can use RSpec filters to exclude performance tests from regular runs. For example, in `spec_helper`:\n\n```ruby\nRSpec.config do |config|\n  config.filter_run_excluding perf: true\nend\n```\n\nAnd then in your example group do:\n\n```ruby\nRSpec.describe ..., :perf do\n  ...\nend\n```\n\nThen you can run groups or examples tagged with `perf`:\n\n```\nrspec --tag perf\n```\n\nAnother option is to simply isolate the performance specs in separate directory such as `spec/performance/...` and add custom rake task to run them.\n\n## 5. Caveats\n\nWhen writing performance tests things to be mindful are:\n\n+ The tests may **potentially be flaky** thus its best to use sensible boundaries:\n  - **too strict** boundaries may cause false positives, making tests fail\n  - **too relaxed** boundaries may also lead to false positives missing actual performance regressions\n+ Generally performance tests will be **slow**, but you may try to avoid _unnecessarily_ slow tests by choosing smaller maximum value for sampling\n\nIf you have any other observations please share them!\n\n## Contributing\n\n1. Fork it ( https://github.com/piotrmurach/rspec-benchmark/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n\n## Code of Conduct\n\nEveryone interacting in the Strings project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/piotrmurach/rspec-benchmark/blob/master/CODE_OF_CONDUCT.md).\n\n## Copyright\n\nCopyright (c) 2016 Piotr Murach. See LICENSE for further details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpiotrmurach%2Frspec-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpiotrmurach%2Frspec-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpiotrmurach%2Frspec-benchmark/lists"}