{"id":15654208,"url":"https://github.com/s3cur3/parameterized_test","last_synced_at":"2025-12-12T00:15:28.971Z","repository":{"id":222029760,"uuid":"755914891","full_name":"s3cur3/parameterized_test","owner":"s3cur3","description":"A utility for defining eminently readable parameterized (or example-based) tests in Elixir","archived":false,"fork":false,"pushed_at":"2025-06-24T13:03:01.000Z","size":157,"stargazers_count":58,"open_issues_count":6,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-29T15:18:18.210Z","etag":null,"topics":["elixir","exunit","testing"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/s3cur3.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-02-11T13:23:17.000Z","updated_at":"2025-06-06T06:56:47.000Z","dependencies_parsed_at":"2024-09-06T19:27:05.214Z","dependency_job_id":"538fe83a-599a-41c3-8140-694b6de2171d","html_url":"https://github.com/s3cur3/parameterized_test","commit_stats":null,"previous_names":["s3cur3/example_test","s3cur3/parameterized_test"],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/s3cur3/parameterized_test","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3cur3%2Fparameterized_test","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3cur3%2Fparameterized_test/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3cur3%2Fparameterized_test/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3cur3%2Fparameterized_test/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/s3cur3","download_url":"https://codeload.github.com/s3cur3/parameterized_test/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/s3cur3%2Fparameterized_test/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262614545,"owners_count":23337288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","exunit","testing"],"created_at":"2024-10-03T12:49:58.622Z","updated_at":"2025-12-12T00:15:28.879Z","avatar_url":"https://github.com/s3cur3.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ParameterizedTest\n\n[![Hex.pm](https://img.shields.io/hexpm/v/parameterized_test)](https://hex.pm/packages/parameterized_test) [![Build and Test](https://github.com/s3cur3/parameterized_test/actions/workflows/elixir-build-and-test.yml/badge.svg)](https://github.com/s3cur3/parameterized_test/actions/workflows/elixir-build-and-test.yml) [![Elixir Quality Checks](https://github.com/s3cur3/parameterized_test/actions/workflows/elixir-quality-checks.yml/badge.svg)](https://github.com/s3cur3/parameterized_test/actions/workflows/elixir-quality-checks.yml) [![Code coverage](https://codecov.io/gh/s3cur3/parameterized_test/graph/badge.svg)](https://codecov.io/gh/s3cur3/parameterized_test)\n\nA utility for defining eminently readable parameterized (or example-based) tests in \nElixir's ExUnit, inspired by [example tests in Cucumber](https://cucumber.io/docs/guides/10-minute-tutorial/?lang=java#using-variables-and-examples).\n\n## What are parameterized tests?\n\nParameterized tests let you define variables along a number of dimensions \nand re-run the same test body (including all `setup`) for each \ncombination of variables.\n\nA simple example:\n\n```elixir\nsetup context do\n  # context.permissions gets set by the param_test below\n  permissions = Map.get(context, :permissions, nil)\n  user = AccountsFixtures.user_fixture(permissions: permissions)\n  %{user: user}\nend\n\nparam_test \"users with editor permissions or better can edit posts\",\n           \"\"\"\n           | permissions | can_edit? | description                     |\n           |-------------|-----------|---------------------------------|\n           | :admin      | true      | Admins have max permissions     |\n           | :editor     | true      | Editors can edit (of course!)   |\n           | :viewer     | false     | Viewers are read-only           |\n           | nil         | false     | Anonymous viewers are read-only |\n           \"\"\",\n           %{user: user, permissions: permissions, can_edit?: can_edit?} do\n  assert Posts.can_edit?(user) == can_edit?, \"#{permissions} permissions should grant edit rights\"\nend\n```\n\nThat test will run 4 times, with the variables from from the table being applied \nto the test's context each time (and therefore being made\navailable to the `setup` handler). These variables are:\n\n- `:permissions` \n- `:can_edit?`\n- the special `:description` variable (see\n  [About test names, and improving debuggability](#about-test-names-and-improving-debuggability)\n  for how this is used)\n\nThus, under the hood this generates four unique tests,\nequivalent to doing something like this:\n\n```elixir\nsetup context do\n  permissions = Map.get(context, :permissions, nil)\n  user = AccountsFixtures.user_fixture{permissions: permissions}\n  %{user: user}\nend\n\nfor {permissions, can_edit?, description} \u003c- [\n        {:admin,  true,  \"Admins have max permissions\"},\n        {:editor, true,  \"Editors can edit (of course!)\"},\n        {:viewer, false, \"Viewers are read-only\"},\n        {nil,     false, \"Anonymous viewers are read-only\"}\n      ] do\n    @permissions permissions\n    @can_edit? can_edit?\n    @description description\n\n    @tag permissions: @permissions\n    @tag can_edit?: @can_edit?\n    @tag description: @description\n    test \"users with at least editor permissions can edit posts — #{@description}\", %{user: user} do\n      assert Posts.can_edit?(user) == @can_edit?\n    end\n  end\nend\n```\n\nAs you can see, even with only 3 variables (just 2 that impact the test semantics!),\nthe `for` comprehension comes with a lot of boilerplate. But the `param_test`\nmacro supports an arbitrary number of variables, so you can describe complex\nbusiness rules like \"users get free shipping if they spend more than $100,\nor if they buy socks, or if they have the right coupon code\":\n\n```elixir\nparam_test \"grants free shipping based on the marketing site's stated policy\",\n            \"\"\"\n            | spending_by_category          | coupon      | ships_free? | description      |\n            |-------------------------------|-------------|-------------|------------------|\n            | %{shoes: 19_99, pants: 29_99} |             | false       | Spent too little |\n            | %{shoes: 59_99, pants: 49_99} |             | true        | Spent over $100  |\n            | %{socks: 10_99}               |             | true        | Socks ship free  |\n            | %{pants: 1_99}                | \"FREE_SHIP\" | true        | Correct coupon   |\n            | %{pants: 1_99}                | \"FOO\"       | false       | Incorrect coupon |\n            \"\"\",\n            %{\n               spending_by_category: spending_by_category,\n               coupon: coupon,\n               ships_free?: ships_free?\n             } do\n  shipping_cost = ShippingCalculator.calculate(spending_by_category, coupon)\n  \n  if ships_free? do\n    assert shipping_cost == 0\n  else\n    assert shipping_cost \u003e 0\n  end\nend\n```\n\nThe package also provides a second macro, `param_feature`, which wraps\nWallaby's `feature` tests the same way `param_test` wraps ExUnit's `test`.\n(While you _can_ use the plain `param_test` macro in a test module that\ncontains `use Wallaby.Feature`, doing so will break some Wallaby features\nincluding screenshot generation on failure.)\n\n## Why parameterized testing?\n\nParameterized testing reduces toil associated with writing tests that cover\na wide variety of different example cases. It also localizes the test logic\ninto a single place, so that at a glance you can see how a number of\ndifferent factors affect the behavior of the system under test.\n\nAs a bonus, a table of examples (with their expected results) often\nmatches how the business communicates the requirements of a system,\nboth internally and to customers—for instance, in a table describing\nshipping costs based on how much a customer spends, where they're\nlocated, whether they've bought a promotional product, etc. This means\nparameterized tests can often be initially created by pulling directly from\na requirements document that your product folks provide, and the\nproduct folks can later read the tests (or at least the parameters table)\nif they want to verify the behavior of the system.\n\n### Parameterized tests versus property tests\n\nParameterized tests are superficially similar to property-based tests.\nBoth allow you to write fewer tests while covering more of your system's\nbehavior. This library is not a replacement for property tests, but\nrather complimentary to them.\n\nThere are a few reasons you might choose to write a parameterized\ntest rather than a property test:\n\n- **When describing policies, not invariants**: Much of a system's\n  business logic comes down to arbitrary choices made by a product team.\n  For instance, there's nothing in the abstract description of a shipping\n  calculator that says buying socks or spending $100 total should grant\n  you free shipping. Those aren't *principles* that every correctly\n  implemented shipping system would implement. Instead, they're choices\n  made by someone (maybe a product manager) which will in all likelihood\n  be fiddled with over time.\n  \n  Contrast that with the classic use cases for property tests: every \n  (correct) implementation of, say, `List.sort/1` will *always* have\n  the dual properties of:\n  \n  1. every element of the input being represented in the output, and \n  2. every element being \"less than\" the element after it.\n\n  These sorting properties are *invariants* of the sorting function,\n  and therefore are quite amenable to property testing.\n- **Ease of writing**: Property tests take a lot of practice to get\n  good at writing. They're often quite time consuming to produce, and\n  even when you think you've adequately described the parameters to\n  the system.\n- **For communication with other stakeholders**: The table of examples\n  in a parameterized test can be made readable by non-programmers (or \n  non-Elixir  programmers), so they can be a good way of showing others\n  in your organization which behaviors of the system you've verified.\n  Because they can compactly express a lot of test cases, they're\n  much more suitable for this than saying \"go read the title of\n  every line in this file that starts with `test`.\"\n- **For verifying the exact scenarios described by other stakeholders**:\n  Sometimes the edges of a particular behavior may be fuzzy—not just\n  to you, but in the business domain as well. Hammering out hard-and-fast\n  rules may not be necessary or worth it, so property tests that exercise\n  the boundaries would be overkill. In contrast, when your product\n  folks produce a document that describes the behavior of particular\n  scenarios, you can encode that in a table and ensure that for the\n  cases that are well-specified, the system behaves correctly.\n\nWhen would you write a property test instead of an example tests?\n\n- When you can specify true invariants about the desired behavior\n- When you want the absolute highest confidence in your code\n- When the correctness of a piece of code is important enough to merit\n  a large time investment in getting the tests right\n- When the system's behavior at the edges is well specified\n\nAnd of course there's nothing wrong with using a mix of normal tests,\nparameterized tests, and property tests for a given piece of functionality.\n\n## Installation and writing your first test\n\n1. Add `parameterized_test` to your `mix.exs` dependencies:\n\n    ```elixir\n    def deps do\n      [\n        {:parameterized_test, \"~\u003e 0.6\", only: [:test]},\n      ]\n    end\n    ```\n2. Run `$ mix deps.get` to download the package\n3. Write your first example test by adding `import ParameterizedTest` \n   to the top of your test module, and using the `param_test` macro.\n\n   You can optionally include a separator between the header and body\n   of the table (like `|--------|-------|`), and a `description` column\n   to improve the errors you get when your test fails (see\n   [About test names, and improving debuggability](#about-test-names-and-improving-debuggability)\n   for more on descriptions).\n\n   The header of your table will be parsed as atoms to pass into your\n   test context. The body cells of the table can be any valid Elixir \n   expression, and empty cells will produce a `nil` value.\n\n   A dummy example:\n\n    ```elixir\n    defmodule MyApp.MyModuleTest do\n      use ExUnit.Case, async: true\n      import ParameterizedTest\n\n      param_test \"behaves as expected\",\n                 \"\"\"\n                 | variable_1       | variable_2           | etc     |\n                 | ---------------- | -------------------- | ------- |\n                 | %{foo: :bar}     | div(19, 3)           | false   |\n                 | \"bip bop\"        | String.upcase(\"foo\") | true    |\n                 | [\"whiz\", \"bang\"] | :ok                  |         |\n                 |                  | nil                  | \"maybe\" |\n                 \"\"\",\n                 %{\n                   variable_1: variable_1,\n                   variable_2: variable_2,\n                   etc: etc\n                 } do\n          assert MyModule.valid_combination?(variable_1, variable_2, etc)\n        end\n    end\n    ```\n\n\n## Debugging failing tests \n\nAs of v0.6.0, when you get a test failure, the backtrace that ExUnit prints will include\nthe line in your file that provided the failing parameters. For instance, consider\nthis test that will fail 100% of the time:\n\n```elixir\nparam_test \"gives the failing parameter row when a test fails\",\n            \"\"\"\n            | should_fail? | description |\n            | false        | \"Works\"     |\n            | true         | \"Breaks\"    |\n            \"\"\",\n            %{should_fail?: should_fail?} do\n  refute should_fail?\nend\n```\n\nWhen you run it, you'll get an ExUnit backtrace that looks like this:\n\n```\n  1) test gives the failing parameter row when a test fails - Breaks (ParameterizedTest.BacktraceTest)\n     test/parameterized_test/backtrace_test.exs:1\n     Expected truthy, got false\n     code: assert not should_fail?\n     stacktrace:\n       test/parameterized_test/backtrace_test.exs:8: (test)\n       test/parameterized_test/backtrace_test.exs:5: ParameterizedTest.BacktraceTest.\"| true         | \\\"Breaks\\\"    |\"/0\n```\n\nUse this line number to figure out which of your parameter rows caused the failing test.\n\n### About test names, and improving debuggability\n\nExUnit requires each test in a module to have a unique name. By default,\nwithout a `description` for the rows in your parameters table, \n`ParameterizedTest` appends a stringified version of the parameters\npassed to your test to the name you give the test. Consider this test:\n\n```elixir\nparam_test \"checks equality\",\n            \"\"\"\n            | val_1 | val_2 |\n            | :a    | :a    |\n            | :b    | :c    |\n            \"\"\",\n            %{val_1: val_1, val_2: val_2} do\n  assert val_1 == val_2\nend\n```\n\nUnder the hood, this produces two tests with the names:\n\n- `\"checks equality ([val_1: :a, val_b: :a])\"`\n- `\"checks equality ([val_1: :b, val_b: :c])\"`\n\nAnd if you ran this test, you'd get an error that looks like this:\n\n```\n  1) test checks equality ([val_1: :b, val_2: :c]) (MyModuleTest)\n     test/my_module_test.exs:4\n     Assertion with == failed\n     code:  assert val_1 == val_2\n     left:  :b\n     right: :c\n     stacktrace:\n       test/my_module_test.exs:11: (test)\n```\n\nYou can improve the names in the failure cases by providing a `description`\ncolumn. When provided, that column will be used in the name. You may want\nto use this to explain *why* this combination of values should produce\nthe expected outcome; for instance:\n\n```elixir\nparam_test \"grants free shipping for spending $99 or more, or with coupon FREE_SHIP\",\n           \"\"\"\n           | total_cents | coupon      | free? | description                 |\n           | ----------- | ----------- | ----- | --------------------------- |\n           | 98_99       |             | false | Spent too little            |\n           | 99_00       |             | true  | Min for free shipping       |\n           | 99_01       |             | true  | Spent more than the minimum |\n           | 1_00        | \"FREE_SHIP\" | true  | Had the right coupon        |\n           | 1_00        | \"FOO\"       | false | Unrecognized coupon         |\n           \"\"\", %{total_cents: total_cents, coupon: coupon, free?: gets_free_shipping?} do\n  shipping_cost = ShippingCalculator.calculate(total_cents, coupon)\n  free_shipping? = shipping_cost == 0\n  assert free_shipping? == gets_free_shipping?\nend\n```\n\nSuppose in your `ShippingCalculator` implementation, you mistakenly set \nthe free shipping threshold to be _greater_ than $99.00, when your web site's\nstate policy was $99 or more. You'd get an error when running this test that\nlooks like this (note the first line ends with \"Spent the min. for free shipping\" from the `description` column):\n\n```\n  1) test grants free shipping for spending $99 or more, or with coupon FREE_SHIP - Spent the min. for free shipping (ShippingCalculatorTest)\n     test/shipping/shipping_calculator_test.exs:34\n     Assertion with == failed\n     code:  assert free_shipping? == gets_free_shipping?\n     left:  false\n     right: true\n     stacktrace:\n       test/shipping/shipping_calculator_test.exs:47: (test)\n```\n\n\n## Objections\n\n### Why not just use the new `:parameterize` feature built into ExUnit in Elixir 1.18?\n\nBoth this package and the\n[new, built-in parameterization](https://hexdocs.pm/ex_unit/main/ExUnit.Case.html#module-parameterized-tests)\ndo similar things: they re-run the same test body with a different set of\nparameters. However, the built-in parameterization works by re-running \nyour entire test module with each parameter set, so it's primarily aimed\nat cases where the tests should work the same regardless of the parameters.\n(The docs give the example of testing the `Registry` module and expecting\nit to behave the same regardless of how many partitions are used.)\n\nIn contrast, the `param_test` macro is designed to use different parameters\non a per-test basis, and it's expected that the parameters will cause different\nbehavior between the test runs (and you'd generally expect to see one column\nthat describes what the results should be).\n\nFinally, of course, there's the format of a `param_test`. The tabular, often\nquite human-friendly format encourages collaboration with less technical\npeople on your team; your product manager may not be able to read a `for`\ncomprehension, but if you link them to a Markdown table on GitHub that shows\nthe test cases you've covered, they can probably make sense of them.\n\n### \"I hate the Markdown table syntax!\"\n\nNo sweat, you don't have to use it. You can instead pass a hand-rolled list of\nparameters to the `param_test` macro, like this:\n\n```elixir\nparam_test \"shipping policy matches the web site\",\n            [\n              # Items in the parameters list can be either maps...\n              %{spending_by_category: %{pants: 29_99}, coupon: \"FREE_SHIP\"},\n              # ...or keyword lists\n              [spending_by_category: %{shoes: 19_99, pants: 29_99}, coupon: nil]\n            ],\n            %{spending_by_category: spending_by_category, coupon: coupon} do\n  ...\nend\n```\n\nJust make sure that each item in the parameters list has the same keys.\n\nThe final option is to pass a path to a *file* that contains your test parameters (we currently support `.md`/`.markdown`, `.csv`, and `.tsv` files), like this:\n\n```elixir\nparam_test \"pull test parameters from a file\",\n            \"test/fixtures/params.md\",\n            %{\n              spending_by_category: spending_by_category,\n              coupon: coupon,\n              gets_free_shipping?: gets_free_shipping?\n            } do\n  ...\nend\n```\n\n## Advanced Configuration\n\n### Use the `~PARAMS` sigil for automated Markdown table formatting\n\nIf you would like some help from `mix format` to align that Markdown table syntax, \nyou can use the optional [`~PARAMS`](https://hexdocs.pm/parameterized_test/ParameterizedTest.Sigil.html#sigil_PARAMS/2) sigil.\n\nFirst, you'll need to update the dependency configuration to include the `:dev` Mix\nenvironment. We want the library to be available when running `mix format`.\n\n```diff\n-  {:parameterized_test, \"~\u003e 0.6\", only: [:test]},\n+  {:parameterized_test, \"~\u003e 0.6\", only: [:dev, :test]},\n```\n\nNext, update your project's `.formatter.exs` file and add \n`ParameterizedTest.Formatter` to the `plugins:` key.\n\n```elixir\n    plugins: [\n      ParameterizedTest.Formatter,\n    ],\n```\n\nFinally, inside your test module, add `import ParameterizedTest.Sigil` to allow use \nof the `~PARAMS` sigil. \n\nWith this sigil in use, any execution of `mix format` will realign your Markdown \ntable syntax. Adding this can be particularly helpful if you collaborate \nwith other developers.\n\nSample usage of the sigil to ensure the formatter will automatically shrink or expand the columns in the future:\n```elixir\nparam_test \"users with editor permissions or better can edit posts\",\n           ~PARAMS\"\"\"\n           | permissions | can_edit? | description                     |\n           |-------------|-----------|---------------------------------|\n           | :admin      | true      | Admins have max permissions     |\n           | :editor     | true      | Editors can edit (of course!)   |\n           | :viewer     | false     | Viewers are read-only           |\n           | nil         | false     | Anonymous viewers are read-only |\n           \"\"\",\n           %{user: user, permissions: permissions, can_edit?: can_edit?} do\n  assert Posts.can_edit?(user) == can_edit?, \"#{permissions} permissions should grant edit rights\"\nend\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3cur3%2Fparameterized_test","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fs3cur3%2Fparameterized_test","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fs3cur3%2Fparameterized_test/lists"}