{"id":16746810,"url":"https://github.com/ohbarye/pbt","last_synced_at":"2025-04-08T09:13:23.392Z","repository":{"id":219560771,"uuid":"749006486","full_name":"ohbarye/pbt","owner":"ohbarye","description":"Property-Based Testing tool for Ruby that supports concurrency with Ractor.","archived":false,"fork":false,"pushed_at":"2024-12-30T12:26:43.000Z","size":202,"stargazers_count":216,"open_issues_count":2,"forks_count":5,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-01T08:41:35.446Z","etag":null,"topics":["parallel","property-based-testing","ractor","ruby","testing"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/pbt","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ohbarye.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-27T10:02:45.000Z","updated_at":"2025-03-14T01:44:41.000Z","dependencies_parsed_at":"2024-05-06T05:26:29.754Z","dependency_job_id":"cade1678-dc77-4720-aeed-fd3267ffa930","html_url":"https://github.com/ohbarye/pbt","commit_stats":{"total_commits":102,"total_committers":4,"mean_commits":25.5,"dds":0.02941176470588236,"last_synced_commit":"d8c99fd3719ea3f5f7231702f60936f1da0a8dc7"},"previous_names":["ohbarye/pbt"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohbarye%2Fpbt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohbarye%2Fpbt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohbarye%2Fpbt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ohbarye%2Fpbt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ohbarye","download_url":"https://codeload.github.com/ohbarye/pbt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247809964,"owners_count":20999816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parallel","property-based-testing","ractor","ruby","testing"],"created_at":"2024-10-13T02:08:23.931Z","updated_at":"2025-04-08T09:13:23.348Z","avatar_url":"https://github.com/ohbarye.png","language":"Ruby","readme":"# Property-Based Testing in Ruby\n\n[![Gem Version](https://badge.fury.io/rb/pbt.svg)](https://rubygems.org/gems/pbt)\n[![Build Status](https://github.com/ohbarye/pbt/actions/workflows/main.yml/badge.svg)](https://github.com/ohbarye/pbt/actions/workflows/main.yml)\n[![RubyDoc](https://img.shields.io/badge/%F0%9F%93%9ARubyDoc-documentation-informational.svg)](https://www.rubydoc.info/gems/pbt)\n\nA property-based testing tool for Ruby with experimental features that allow you to run test cases in parallel.\n\nPBT stands for Property-Based Testing.\n\nAs for the results of the parallelization experiment, please refer the talk at RubyKaigi 2024: [Unlocking Potential of Property Based Testing with Ractor](https://rubykaigi.org/2024/presentations/ohbarye.html).\n\n## What's Property-Based Testing?\n\nProperty-Based Testing is a testing methodology that focuses on the properties a system should always satisfy, rather than checking individual examples. Instead of writing tests for predefined inputs and outputs, PBT allows you to specify the general characteristics that your code should adhere to and then automatically generates a wide range of inputs to verify these properties.\n\nThe key benefits of property-based testing include the ability to cover more edge cases and the potential to discover bugs that traditional example-based tests might miss. It's particularly useful for identifying unexpected behaviors in your code by testing it against a vast set of inputs, including those you might not have considered.\n\nFor a more in-depth understanding of Property-Based Testing, please refer to external resources.\n\n- Original ideas\n  - [Property-based testing of privileged programs](https://ieeexplore.ieee.org/document/367311) (1994)\n  - [Property-based testing: a new approach to testing for assurance](https://dl.acm.org/doi/abs/10.1145/263244.263267) (1997)\n  - [QuickCheck: a lightweight tool for random testing of Haskell programs](https://dl.acm.org/doi/10.1145/351240.351266) (2000)\n- Rather new introductory resources\n  - Fred Hebert's book [Property-Based Testing With PropEr, Erlang and Elixir](https://propertesting.com/).\n  - [fast-check - Why Property-Based?](https://fast-check.dev/docs/introduction/why-property-based/)\n\n## Installation\n\nAdd this line to your application's Gemfile and run `bundle install`.\n\n```ruby\ngem 'pbt'\n```\n\nOff course you can install with `gem intstall pbt`.\n\n## Basic Usage\n\n### Simple property\n\n```ruby\n# Let's say you have your own sort method.\ndef sort(array)\n  return array if array.size \u003c= 2 # Here's a bug! It should be 1.\n  pivot, *rest = array\n  left, right = rest.partition { |n| n \u003c= pivot }\n  sort(left) + [pivot] + sort(right)\nend\n\nPbt.assert do\n  # The given block is executed 100 times with different arrays with random numbers.\n  # Besides, if you set `worker: :ractor` option to `assert` method, it runs in parallel using Ractor.\n  Pbt.property(Pbt.array(Pbt.integer)) do |numbers|\n    result = sort(numbers)\n    result.each_cons(2) do |x, y|\n      raise \"Sort algorithm is wrong.\" unless x \u003c= y\n    end\n  end\nend\n\n# If the method has a bug, the test fails and it reports a minimum counterexample.\n# For example, the sort method doesn't work for [0, -1].\n#\n# Pbt::PropertyFailure:\n#   Property failed after 23 test(s)\n#   seed: 43738985293126714007411539287084402325\n#   counterexample: [0, -1]\n#   Shrunk 40 time(s)\n#   Got RuntimeError: Sort algorithm is wrong.\n```\n\n### Explain The Snippet\n\nThe above snippet is very simple but contains the basic components.\n\n#### Runner\n\n`Pbt.assert` is the runner. The runner interprets and executes the given property. `Pbt.assert` takes a property and runs it multiple times. If the property fails, it tries to shrink the input that caused the failure.\n\n#### Property\n\nThe snippet above declared a property by calling `Pbt.property`. The property describes the following:\n\n1. What the user wants to evaluate. This corresponds to the block (let's call this `predicate`) enclosed by `do` `end`\n2. How to generate inputs for the predicate — using `Arbitrary`\n\nThe `predicate` block is a function that directly asserts, taking values generated by `Arbitrary` as input.\n\n#### Arbitrary\n\nArbitrary generates random values. It is also responsible for shrinking those values if asked to shrink a failed value as input.\n\nHere, we used only one type of arbitrary, `Pbt.integer`. There are many other built-in arbitraries, and you can create a variety of inputs by combining existing ones.\n\n#### Shrink\n\nIn PBT, If a test fails, it attempts to shrink the case that caused the failure into a form that is easier for humans to understand.\nIn other words, instead of stopping the test itself the first time it fails and reporting the failed value, it tries to find the minimal value that causes the error.\n\nWhen there is a test that fails when given an even number, a counterexample of `[0, -1]` is simpler and easier to understand than any complex example like `[-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]`.\n\n### Arbitrary\n\nThere are many built-in arbitraries in `Pbt`. You can use them to generate random values for your tests. Here are some representative arbitraries.\n\n#### Primitives\n\n```ruby\nrng = Random.new\n\nPbt.integer.generate(rng)                  # =\u003e 42\nPbt.integer(min: -1, max: 8).generate(rng) # =\u003e Integer between -1 and 8\n\nPbt.symbol.generate(rng)                   # =\u003e :atq\n\nPbt.ascii_char.generate(rng)               # =\u003e \"a\"\nPbt.ascii_string.generate(rng)             # =\u003e \"aagjZfao\"\n\nPbt.boolean.generate(rng)                  # =\u003e true or false\nPbt.constant(42).generate(rng)             # =\u003e 42 always\n```\n\n#### Composites\n\n```ruby\nrng = Random.new\n\nPbt.array(Pbt.integer).generate(rng)                        # =\u003e [121, -13141, 9825]\nPbt.array(Pbt.integer, max: 1, empty: true).generate(rng)   # =\u003e [] or [42] etc.\n\nPbt.tuple(Pbt.symbol, Pbt.integer).generate(rng)            # =\u003e [:atq, 42]\n\nPbt.fixed_hash(x: Pbt.symbol, y: Pbt.integer).generate(rng) # =\u003e {x: :atq, y: 42}\nPbt.hash(Pbt.symbol, Pbt.integer).generate(rng)             # =\u003e {atq: 121, ygab: -1142}\n\nPbt.one_of(:a, 1, 0.1).generate(rng)                        # =\u003e :a or 1 or 0.1\n````\n\nSee [ArbitraryMethods](https://github.com/ohbarye/pbt/blob/main/lib/pbt/arbitrary/arbitrary_methods.rb) module for more details.\n\n## What if property-based tests fail?\n\nOnce a test fails it's time to debug. `Pbt` provides some features to help you debug.\n\n### How to reproduce\n\nWhen a test fails, you'll see a message like below.\n\n```text\nPbt::PropertyFailure:\n  Property failed after 23 test(s)\n  seed: 43738985293126714007411539287084402325\n  counterexample: [0, -1]\n  Shrunk 40 time(s)\n  Got RuntimeError: Sort algorithm is wrong.\n  # and backtraces\n```\n\nYou can reproduce the failure by passing the seed to `Pbt.assert`.\n\n```ruby\nPbt.assert(seed: 43738985293126714007411539287084402325) do\n  Pbt.property(Pbt.array(Pbt.integer)) do |number|\n    # your test\n  end\nend\n```\n\n### Verbose mode\n\nYou may want to know which values pass and which values fail. You can enable verbose mode by passing `verbose: true` to `Pbt.assert`.\n\n```ruby\nPbt.assert(verbose: true) do\n  Pbt.property(Pbt.array(Pbt.integer)) do |numbers|\n    # your failed test\n  end\nend\n```\n\nThe verbose mode prints the results of each tested values.\n\n```text\nEncountered failures were:\n- [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]\n- [310864, 856411, -304517, 86613, -78231]\n- [-304517, 86613, -78231]\n(snipped for README)\n- [0, -3]\n- [0, -2]\n- [0, -1]\n\nExecution summary:\n. × [-897860, -930517, 577817, -16302, 310864, 856411, -304517, 86613, -78231]\n. . √ [-897860, -930517, 577817, -16302, 310864]\n. . √ [-930517, 577817, -16302, 310864, 856411]\n. . √ [577817, -16302, 310864, 856411, -304517]\n. . √ [-16302, 310864, 856411, -304517, 86613]\n. . × [310864, 856411, -304517, 86613, -78231]\n(snipped for README)\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [-2]\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ []\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . × [0, -1]\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [0]\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [-1]\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ []\n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . √ [0, 0]\n```\n\n## Configuration\n\nYou can configure `Pbt` by calling `Pbt.configure` before running tests.\n\n```ruby\nPbt.configure do |config|\n  # Whether to print verbose output. Default is `false`.\n  config.verbose = false\n\n  # The concurrency method to use. `:ractor` and `:none` are supported. Default is `:none`.\n  config.worker = :none\n\n  # The number of runs to perform. Default is `100`.\n  config.num_runs = 100\n\n  # The seed to use for random number generation.\n  # It's useful to reproduce failed test with the seed you'd pick up from failure messages. Default is a random seed.\n  config.seed = 42\n\n  # Whether to report exceptions in threads.\n  # It's useful to suppress error logs on Ractor that reports many errors. Default is `false`.\n  config.thread_report_on_exception = false\nend\n```\n\nOr, you can pass the configuration to `Pbt.assert` as an argument.\n\n```ruby\nPbt.assert(num_runs: 100, seed: 42) do\n  # ...\nend\n```\n\n## Concurrency methods\n\nOne of the key features of `Pbt` is its ability to rapidly execute test cases in parallel or concurrently, using a large number of values (by default, `100`) generated by `Arbitrary`.\n\nFor concurrent processing, you can specify `:ractor` using the `worker` option. Alternatively, choose `:none` for serial execution.\n\nBe aware that the performance of each method depends on the test subject. For example, if the test subject is CPU-bound, `:ractor` may be the best choice. Otherwise, `:none` shall be the best choice for most cases. See [benchmarks](benchmark/README.md).\n\n### Ractor\n\n`:ractor` worker is useful for test cases that are CPU-bound. But it's experimental and has some limitations as described below. If you encounter any issues due to those limitations, consider falling back to `:none`.\n\n```ruby\nPbt.assert(worker: :ractor) do\n  Pbt.property(Pbt.integer) do |n|\n    # ...\n  end\nend\n```\n\n#### Limitation\n\nPlease note that Ractor support is an experimental feature of this gem. Due to Ractor's limitations, you may encounter some issues when using it.\n\nFor example, you cannot access anything out of block.\n\n```ruby\na = 1\n\nPbt.assert(worker: :ractor) do\n  Pbt.property(Pbt.integer) do |n|\n    # You cannot access `a` here because this block is executed in a Ractor and it doesn't allow implicit sharing of objects.\n    a + n # =\u003e Ractor::RemoteError (can not share object between ractors)\n  end\nend\n```\n\nYou cannot use any methods provided by test frameworks like `expect` or `assert` because they are not available in a Ractor.\n\n```ruby\nit do\n  Pbt.assert(worker: :ractor) do\n    Pbt.property(Pbt.integer) do |n|\n      # This is not possible because `self` if a Ractor here.\n      expect(n).to be_an(Integer) # =\u003e Ractor::RemoteError (cause by NoMethodError for `expect` or `be_an`)\n    end\n  end\nend\n```\n\n### None\n\nFor most cases, `:none` is the best choice. It runs tests sequentially but most test cases finishes within a reasonable time.\n\n```ruby\nPbt.assert(worker: :none) do\n  Pbt.property(Pbt.integer) do |n|\n    # ...\n  end\nend\n```\n\n## TODOs\n\nOnce this project finishes the following, we will release v1.0.0.\n\n- [x] Implement basic primitive arbitraries\n- [x] Implement composite arbitraries\n- [x] Support shrinking\n- [x] Support multiple concurrency methods\n  - [x] Ractor\n  - [x] Process (dropped)\n  - [x] Thread (dropped)\n  - [x] None (Run tests sequentially)\n- [x] Documentation\n  - [x] Add better examples\n  - [x] Arbitrary usage\n  - [x] Configuration\n- [x] Benchmark\n- [x] Rich report by verbose mode\n- [x] (Partially) Allow to use expectations and matchers provided by test framework in Ractor. (dropped)\n  - It'd be so hard to pass assertions like `expect`, `assert` to a Ractor.\n- [ ] Implement frequency arbitrary\n- [ ] Statistics feature to aggregate generated values\n- [ ] Decide DSL\n- [ ] Try Fiber\n- [ ] Stateful property-based testing\n\n## Development\n\n### Setup\n\n```shell\nbin/setup\nbundle exec rake # Run tests and lint at once\n```\n\n### Test\n\n```shell\nbundle exec rspec\n```\n\n### Lint\n\n```shell\nbundle exec rake standard:fix\n```\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/ohbarye/pbt. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/ohbarye/pbt/blob/master/CODE_OF_CONDUCT.md).\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).\n\n## Credits\n\nThis project draws a lot of inspiration from other testing tools, namely\n\n- [fast-check](https://fast-check.dev/)\n- [Loupe](https://github.com/vinistock/loupe)\n- [RSpec](https://github.com/rspec/rspec)\n- [Minitest](https://github.com/seattlerb/minitest)\n- [Parallel](https://github.com/grosser/parallel)\n- [PropCheck for Ruby](https://github.com/Qqwy/ruby-prop_check)\n- [PropCheck for Elixir](https://github.com/alfert/propcheck)\n\n## Code of Conduct\n\nEveryone interacting in the Pbt project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/ohbarye/pbt/blob/master/CODE_OF_CONDUCT.md).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fohbarye%2Fpbt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fohbarye%2Fpbt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fohbarye%2Fpbt/lists"}