{"id":31588608,"url":"https://github.com/wende/ab","last_synced_at":"2026-01-20T16:58:14.023Z","repository":{"id":317855701,"uuid":"1069098138","full_name":"wende/ab","owner":"wende","description":"Automatically compare two implementations of the same problem with property-based testing and performance benchmarks.","archived":false,"fork":false,"pushed_at":"2025-10-03T14:16:45.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-03T14:28:12.762Z","etag":null,"topics":["ab-testing","elixir","elixir-lang","property-testing","quickcheck"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wende.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-03T11:59:02.000Z","updated_at":"2025-10-03T14:15:16.000Z","dependencies_parsed_at":"2025-10-03T14:29:21.752Z","dependency_job_id":"5888773a-48a0-4f5d-93c3-e4d08125bb5c","html_url":"https://github.com/wende/ab","commit_stats":null,"previous_names":["wende/ab"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/wende/ab","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wende%2Fab","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wende%2Fab/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wende%2Fab/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wende%2Fab/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wende","download_url":"https://codeload.github.com/wende/ab/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wende%2Fab/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278315198,"owners_count":25966775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-04T02:00:05.491Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ab-testing","elixir","elixir-lang","property-testing","quickcheck"],"created_at":"2025-10-06T02:11:26.846Z","updated_at":"2026-01-20T16:58:14.017Z","avatar_url":"https://github.com/wende.png","language":"Elixir","readme":"# AB\n\n[![Hex.pm](https://img.shields.io/hexpm/v/ab.svg)](https://hex.pm/packages/ab)\n[![Hex Docs](https://img.shields.io/badge/hex-docs-lightgreen.svg)](https://hexdocs.pm/ab/)\n[![Hex.pm Downloads](https://img.shields.io/hexpm/dt/ab.svg)](https://hex.pm/packages/ab)\n[![GitHub CI](https://github.com/wende/ab/workflows/Elixir%20CI/badge.svg)](https://github.com/wende/ab/actions)\n[![License](https://img.shields.io/hexpm/l/ab.svg)](https://github.com/wende/ab/blob/main/LICENSE)\n\n**Automatically compare two implementations of the same problem with property-based testing and performance benchmarks.**\n\nAB is an Elixir library that makes it effortless to verify that two implementations of the same function behave identically, while also comparing their performance characteristics. Perfect for refactoring, algorithm optimization, and A/B testing different approaches.\n\n## Why AB?\n\nWhen you have two implementations of the same function:\n- **Refactoring** - Ensure your optimized version produces identical results\n- **Algorithm comparison** - Compare different algorithms solving the same problem\n- **Migration** - Verify new code matches legacy behavior exactly\n- **Learning** - Understand tradeoffs between different approaches\n\nAB automatically generates property tests from your typespecs and runs comprehensive comparisons.\n\n## Features\n\n✅ **Automatic property test generation** from function typespecs  \n✅ **Side-by-side comparison** of two implementations  \n✅ **Performance benchmarking** with detailed statistics  \n✅ **Invalid input testing** to verify error handling  \n✅ **Type consistency validation** between specs and implementations  \n✅ **Mix task** for testing standalone Elixir files  \n✅ **Zero boilerplate** - just add macros to your tests\n\n## Installation\n\nAdd `ab` to your `mix.exs` dependencies:\n\n```elixir\ndef deps do\n  [\n    {:ab, \"~\u003e 0.1.0\"}\n  ]\nend\n```\n\n## Quick Start\n\n### 1. Define two implementations with identical typespecs\n\n```elixir\ndefmodule Math do\n  # Implementation A: iterative\n  @spec factorial_iterative(non_neg_integer()) :: pos_integer()\n  def factorial_iterative(n), do: factorial_iter(n, 1)\n  \n  defp factorial_iter(0, acc), do: acc\n  defp factorial_iter(n, acc), do: factorial_iter(n - 1, n * acc)\n\n  # Implementation B: recursive\n  @spec factorial_recursive(non_neg_integer()) :: pos_integer()\n  def factorial_recursive(0), do: 1\n  def factorial_recursive(n), do: n * factorial_recursive(n - 1)\nend\n```\n\n### 2. Compare them automatically\n\n```elixir\ndefmodule MathTest do\n  use ExUnit.Case\n  use ExUnitProperties\n  import AB\n\n  # Automatically test both implementations produce identical results\n  compare_test {Math, :factorial_iterative}, {Math, :factorial_recursive}\n\n  # Benchmark performance differences\n  benchmark_test {Math, :factorial_iterative}, {Math, :factorial_recursive}\n\n  # Test each implementation matches its typespec\n  property_test Math, :factorial_iterative\n  property_test Math, :factorial_recursive\nend\n```\n\nThat's it! AB will:\n- Generate random test data matching your typespec\n- Verify both functions produce identical outputs\n- Compare performance with detailed statistics\n- Validate outputs match the declared return type\n\n## Core Macros\n\n### `compare_test/2` - Verify Identical Behavior\n\nGenerates property tests proving two implementations produce identical results:\n\n```elixir\n# Basic comparison\ncompare_test {ModuleA, :function}, {ModuleB, :function}\n\n# With verbose logging\ncompare_test {ModuleA, :function}, {ModuleB, :function}, verbose: true\n```\n\nThe macro will:\n1. Extract and compare typespecs (must be identical)\n2. Generate test data matching the input types\n3. Run both functions on the same inputs\n4. Assert outputs are identical\n5. Validate outputs match the return type\n\n**Example output:**\n```\nproperty factorial_iterative and factorial_recursive produce identical results\n  ✓ 100 successful comparison runs\n✓ factorial_iterative and factorial_recursive produce identical results (1.2ms)\n```\n\n### `benchmark_test/2` - Compare Performance\n\nGenerates benchmarks comparing two implementations:\n\n```elixir\n# Basic benchmark\nbenchmark_test {ModuleA, :function}, {ModuleB, :function}\n\n# Custom timing\nbenchmark_test {ModuleA, :function}, {ModuleB, :function},\n  time: 5,           # 5 seconds of benchmarking\n  memory_time: 2     # 2 seconds of memory profiling\n```\n\n**Example output:**\n```\n=== Benchmarking Math.factorial_iterative vs Math.factorial_recursive ===\n\nName                           ips        average  deviation         median         99th %\nMath.factorial_iterative    1.23 M        0.81 μs   ±612.45%        0.75 μs        1.12 μs\nMath.factorial_recursive    0.98 M        1.02 μs   ±587.32%        0.96 μs        1.35 μs\n\nComparison:\nMath.factorial_iterative    1.23 M\nMath.factorial_recursive    0.98 M - 1.26x slower +0.21 μs\n```\n\n### `property_test/2` - Validate Against Typespec\n\nAutomatically generates property tests from function typespecs:\n\n```elixir\n# Basic property test\nproperty_test MyModule, :my_function\n\n# With verbose logging\nproperty_test MyModule, :my_function, verbose: true\n```\n\nThe macro will:\n1. Parse the function's `@spec` declaration\n2. Generate appropriate test data for all input types\n3. Call the function with generated inputs\n4. Validate outputs match the declared return type\n5. Test type consistency between `@type` and `@spec`\n\n**Supported types:**\n- Basic: `integer()`, `float()`, `number()`, `boolean()`, `atom()`, `binary()`, `bitstring()`, `String.t()`, `charlist()`, `nil`, `iodata`, `no_return`\n- Collections: `list(type)`, `tuple({type1, type2})`, `map()`, `keyword()`, `keyword(type)`\n- Maps: `%{key =\u003e value}`, `%{required(:key) =\u003e type}`, `%{optional(:key) =\u003e type}` (optional fields don't cause validation failures)\n- Functions: `(arg_type -\u003e return_type)`, `(arg1, arg2 -\u003e return)`, `(-\u003e return)` for callbacks and higher-order functions\n- Ranges: `0..100`, `pos_integer()`, `non_neg_integer()`, `neg_integer()`\n- Structs: Custom struct types with `@type t :: %__MODULE__{...}`\n- Union types: `integer() | String.t()`\n- Literals: Specific atom or integer values (e.g., `:ok`, `42`)\n- Generic: `any()`, `term()`\n- Complex: Nested structures, remote types\n\n**Validated against:** Successfully parses all typespecs from real-world libraries like Jason\n\n**Important notes:**\n- **Maps:** Optional fields and extra keys are properly handled - only required fields must be present.\n- **Functions:** Generated functions have correct arity and return correct types, but are \"constant functions\" that ignore their arguments. This still validates that tested functions accept and call function arguments correctly, but doesn't verify the lambda's internal logic.\n\n### `robust_test/2` - Verify Error Handling\n\nTests that functions properly reject invalid inputs:\n\n```elixir\n# Test invalid input handling\nrobust_test MyModule, :my_function\n\n# With verbose logging\nrobust_test MyModule, :my_function, verbose: true\n```\n\nThis generates inputs that **don't** match the typespec and verifies the function either:\n- Raises an appropriate exception\n- Has guards that prevent type mismatches\n\nGreat for ensuring functions fail gracefully rather than producing garbage output.\n\n## Complete Example\n\n```elixir\ndefmodule Sum do\n  # Implementation A: Enum.sum\n  @spec sum_builtin([integer()]) :: integer()\n  def sum_builtin(list), do: Enum.sum(list)\n\n  # Implementation B: manual recursion\n  @spec sum_recursive([integer()]) :: integer()\n  def sum_recursive([]), do: 0\n  def sum_recursive([head | tail]), do: head + sum_recursive(tail)\nend\n\ndefmodule SumTest do\n  use ExUnit.Case\n  use ExUnitProperties\n  import AB\n\n  describe \"Sum implementations\" do\n    # Verify both produce identical results\n    compare_test {Sum, :sum_builtin}, {Sum, :sum_recursive}\n\n    # Compare performance\n    benchmark_test {Sum, :sum_builtin}, {Sum, :sum_recursive}\n\n    # Validate each against typespec\n    property_test Sum, :sum_builtin\n    property_test Sum, :sum_recursive\n\n    # Test error handling\n    robust_test Sum, :sum_builtin\n    robust_test Sum, :sum_recursive\n  end\nend\n```\n\n**Output:**\n```\nSumTest\n  Sum implementations\n    property sum_builtin and sum_recursive produce identical results\n      ✓ 100 successful comparison runs\n    ✓ sum_builtin and sum_recursive produce identical results (1.8ms)\n    \n    property sum_builtin satisfies its typespec\n      ✓ 100 successful property test runs\n    ✓ sum_builtin satisfies its typespec (2.1ms)\n    ✓ sum_builtin type consistency validation (0.1ms)\n    \n    property sum_recursive satisfies its typespec\n      ✓ 100 successful property test runs\n    ✓ sum_recursive satisfies its typespec (2.4ms)\n    ✓ sum_recursive type consistency validation (0.1ms)\n    \n    property sum_builtin properly rejects invalid input\n      ✓ 100 successful invalid input test runs\n    ✓ sum_builtin properly rejects invalid input (124.3ms)\n    \n    property sum_recursive properly rejects invalid input\n      ✓ 100 successful invalid input test runs\n    ✓ sum_recursive properly rejects invalid input (127.8ms)\n    \n    test benchmark sum_builtin vs sum_recursive\n    === Benchmarking Sum.sum_builtin vs Sum.sum_recursive ===\n    Name                   ips        average  deviation\n    Sum.sum_builtin     1.45 M        0.69 μs   ±652.34%\n    Sum.sum_recursive   0.87 M        1.15 μs   ±723.12%\n    \n    Comparison:\n    Sum.sum_builtin     1.45 M\n    Sum.sum_recursive   0.87 M - 1.67x slower +0.46 μs\n    ✓ benchmark sum_builtin vs sum_recursive (7503.5ms)\n\nFinished in 7.9 seconds\n8 properties, 1 test, 0 failures\n```\n\n## API Functions\n\nFor manual testing and custom scenarios:\n\n### `AB.get_function_spec/2`\n\nExtract typespec information:\n\n```elixir\n{:ok, {input_types, output_type}} =\n  AB.get_function_spec(MyModule, :my_function)\n```\n\n### `AB.types_equivalent?/2`\n\nCompare two type specifications:\n\n```elixir\nAB.types_equivalent?(type1, type2)\n# =\u003e true | false\n```\n\n### `AB.infer_result_type/1`\n\nGet detailed type information from a value:\n\n```elixir\nAB.infer_result_type([1, 2, 3])\n# =\u003e \"list(integer())\"\n\nAB.infer_result_type(%{name: \"Alice\", age: 30})\n# =\u003e \"%{age: integer(), name: binary()}\"\n\nAB.infer_result_type({:ok, true})\n# =\u003e \"{atom(), boolean()}\"\n\nAB.infer_result_type([])\n# =\u003e \"list(term())\"  # unknown element type\n\nAB.infer_result_type([1, \"a\"])\n# =\u003e \"list(term())\"  # inconsistent types\n```\n\n## Real-World Examples\n\n### Refactoring for Performance\n\n```elixir\n# Compare old vs new implementation\ncompare_test {Parser, :parse_legacy}, {Parser, :parse_optimized}\nbenchmark_test {Parser, :parse_legacy}, {Parser, :parse_optimized}\n```\n\n### Algorithm Comparison\n\n```elixir\n# Test different search algorithms\ncompare_test {Search, :binary_search}, {Search, :interpolation_search}\n```\n\n### Data Encoding Comparison\n\n```elixir\n# Compare JSON encoding libraries\ncompare_test {Encoder, :encode_with_jason}, {Encoder, :encode_with_poison}\n```\n\n## Mix Task\n\nYou can test standalone Elixir files without setting up a full test suite using the `mix ab.test` task:\n\n```bash\n# Test a single file\nmix ab.test path/to/file.ex\n\n# Test with verbose output\nmix ab.test path/to/file.ex --verbose\n```\n\nThe task will:\n1. Compile the specified Elixir file\n2. Extract the module from the compiled code\n3. Find all exported functions with typespecs\n4. Run AB.property_test on each function\n\nExample:\n\n```bash\n$ mix ab.test lib/my_module.ex\nTesting file: lib/my_module.ex\nRunning property tests for module: MyModule\nFound 3 functions with typespecs\n\n=== Running property tests for 3 functions ===\n\nTesting MyModule.add\n  ✓ 100 successful property test runs\n\nTesting MyModule.multiply\n  ✓ 100 successful property test runs\n\nTesting MyModule.divide\n  ✓ 100 successful property test runs\n\n✓ All property tests completed\n```\n\n## Dependencies\n\n- **stream_data** - Property-based testing and data generation\n- **benchee** - Performance benchmarking\n- **ex_unit** - Elixir's built-in test framework\n\n## Contributing\n\nContributions are welcome! Please:\n\n1. Fork the repository\n2. Create a feature branch\n3. Add tests for new functionality\n4. Submit a pull request\n\n## License\n\nMIT License - see LICENSE file for details\n\n## Credits\n\nBuilt with ❤️ using:\n- [StreamData](https://github.com/whatyouhide/stream_data) by Andrea Leopardi\n- [Benchee](https://github.com/bencheeorg/benchee) by Tobias Pfeiffer\n- Inspired by QuickCheck and property-based testing\n\n---\n\n**Start comparing your implementations today!** 🚀\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwende%2Fab","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwende%2Fab","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwende%2Fab/lists"}