{"id":18677239,"url":"https://github.com/elixir-toniq/alchemy","last_synced_at":"2025-12-11T23:43:24.395Z","repository":{"id":37820098,"uuid":"51881796","full_name":"elixir-toniq/alchemy","owner":"elixir-toniq","description":"Perform experiments in production","archived":false,"fork":false,"pushed_at":"2023-03-30T11:01:24.000Z","size":81,"stargazers_count":83,"open_issues_count":2,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-05T09:23:19.389Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elixir-toniq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-17T00:35:09.000Z","updated_at":"2025-05-17T21:37:26.000Z","dependencies_parsed_at":"2024-11-07T09:36:02.495Z","dependency_job_id":null,"html_url":"https://github.com/elixir-toniq/alchemy","commit_stats":null,"previous_names":["keathley/alchemy"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/elixir-toniq/alchemy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Falchemy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Falchemy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Falchemy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Falchemy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elixir-toniq","download_url":"https://codeload.github.com/elixir-toniq/alchemy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Falchemy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279000709,"owners_count":26082895,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-08T02:00:06.501Z","response_time":56,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T09:33:01.427Z","updated_at":"2025-10-09T08:07:11.779Z","avatar_url":"https://github.com/elixir-toniq.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Alchemy\n\n[![Hex.pm](https://img.shields.io/hexpm/v/alchemy.svg?style=flat-square)](https://hex.pm/packages/alchemy)\n[![Hex.pm](https://img.shields.io/hexpm/dt/alchemy.svg?style=flat-square)](https://hex.pm/packages/alchemy)\n\nSafely perform refactoring experiments in production.\n\n---\n\n## Installation\n\n``` elixir\ndef deps do\n  [{:alchemy, \"~\u003e 0.2.0\"}]\nend\n```\n\n## Perform some experiments\n\nLets say that you have a controller that returns a list of users, and you want to change how that list of users is fetched from the database. Unit tests can help you but they can only account for examples that you've currently considered. Alchemy allows you to test your new code live in production.\n\n```elixir\ndefmodule MyApp.UserController do\n  alias Alchemy.Experiment\n\n  def index(conn) do\n    users =\n      Experiment.new(\"users-query\")\n      |\u003e Experiment.control(\u0026old_slow_query/0)\n      |\u003e Experiment.candidate(\u0026new_fast_query/0)\n      |\u003e Experiment.run\n\n    render(conn, \"index.json\", users: users)\n  end\n\n  defp old_query do\n    # ...\n  end\n\n  defp new_query do\n    # ...\n  end\nend\n```\n\nBoth the control and the candidate are randomized. Once all of the observations\nhave been made the control is returned so that its easy to continue pipelining\nwith other functions. Along with this Alchemy does a few other things:\n\n* Compares the results of the control against the candidates looking for mismatches\n* Measures the execution time of both the control and the candidates\n* Captures and records any errors thrown by the candidates\n* Publishes all of these results\n\n### Publish results\n\nThe experiment is now running but the results aren't being published. In order\nto publish the results We can pass a publisher function to our experiment.\n\n``` elixir\ndefmodule MyApp.UserController do\n  require Logger\n\n  alias Alchemy.{Experiment, Result}\n\n  def index(conn) do\n    users =\n      Experiment.new(\"users-query\")\n      |\u003e Experiment.control(\u0026old_slow_query/0)\n      |\u003e Experiment.candidate(\u0026new_fast_query/0)\n      |\u003e Experiment.publisher(\u0026publish/1)\n      |\u003e Experiment.run\n\n    render(conn, \"index.json\", users: users)\n  end\n\n  def publish(result=%Alchemy.Result{}) do\n    name      = result.name\n    control   = result.control\n    candidate = hd(result.candidates)\n    matched   = Result.matched?(result)\n\n    Logger.debug \"\"\"\n    Test: #{name}\n    Match?: #{!Result.mismatched?(result)}\n    Control - value: #{control.value} | duration: #{control.duration}\n    Candidate - value: #{candidate.value} | duration: #{candidate.duration}\n    \"\"\"\n  end\nend\n```\n\nIf you want to share your publishing logic across multiple experiments then we\ncan pass a module as the producer. Alchemy will assume that there is a `publish/1`\nfunction available on the specified module.\n\n```elixir\ndef index(conn) do\n  users =\n    Experiment.new(\"users-query\")\n    |\u003e Experiment.control(\u0026old_slow_query/0)\n    |\u003e Experiment.candidate(\u0026new_fast_query/0)\n    |\u003e Experiment.publisher(Publisher)\n    |\u003e Experiment.run\nend\n\ndefmodule Publisher do\n  alias Alchemy.Result\n\n  def publish(%{name: name} = result) do\n    Statix.timing(\"alchemy.#{name}.control.duration.ms\", result.control.duration)\n\n    candidate_duration =\n      result.candidates\n      |\u003e Enum.at(0)\n      |\u003e Map.get(:duration)\n\n    Statix.timing(\"alchemy.#{name}.candidate.duration.ms\", candidate_duration)\n\n    # and counts for match/ignore/mismatch:\n    cond do\n      Result.matched?(result) -\u003e\n        Statix.increment(\"alchemy.#{name}.matched.total\")\n\n      Result.ignored?(result) -\u003e\n        Statix.increment(\"alchemy.#{name}.ignored.total\")\n\n      true -\u003e\n        Statix.increment(\"alchemy.#{name}.mismatched.total\")\n        store_result(result)\n    end\n  end\n\n  # Store final results in a public ets table for analysis\n  defp store_result(result) do\n    payload = %{\n      name: result.name,\n      control: observation_payload(result.control),\n      candidate: observation_payload(Enum.at(result.candidates, 0))\n    }\n\n    :ets.insert(:alchemy, {result.name, payload})\n  end\n\n  # If this observation raised an error then store the error\n  # otherwise store the \"cleaned\" value.\n  defp observation_payload(observation) do\n    cond do\n      Result.raised?(observation) -\u003e\n        %{\n          error: observation.error,\n          stacktrace: observation.stacktrace\n        }\n\n      true -\u003e\n        %{value: observation.cleaned_value}\n    end\n  end\nend\n```\n\n### Multiple Candidates\n\nIt's possible to create experiments with multiple candidates:\n\n``` elixir\ndef some_query do\n  experiment(\"test multiple candidates\")\n  |\u003e control(\u0026old_query/0)\n  |\u003e candidate(\u0026foo_query/0)\n  |\u003e candidate(\u0026bar_query/0)\n  |\u003e candidate(\u0026baz_query/0)\n  |\u003e run\nend\n```\n\n### Comparing results\n\nBy default alchemy compares results with `==`. You can override this by supplying your own comparator:\n\n``` elixir\ndef user_name do\n  experiment(\"test name is correct\")\n  |\u003e control(fn -\u003e %{id: 1, name: \"Alice\"} end)\n  |\u003e candidate(fn -\u003e %{id: 2, name: \"Alice\"} end)\n  |\u003e comparator(fn(control, candidate) -\u003e control.name == candidate.name end)\n  |\u003e run\nend\n```\n\n### Cleaning values\n\nOften you won't want to publish the full value that is returned from the observation.\nFor instance if you're comparing 2 large structs you may just want to store the\n`id` fields so you can analysize them later. You can pass a cleaner function to\nyour experiment to define how these cleaned values are created (by default the\ncleaner is an identity function).\n\n```elixir\ndef users do\n  experiment(\"user-list\")\n  |\u003e control(fn -\u003e Repo.all(User) end)\n  |\u003e candidate(fn -\u003e UserContext.list() end)\n  |\u003e clean(fn user -\u003e user.id end)\n  |\u003e run\nend\n```\n\n### Ignoring results\n\nIf you have certain scenarios that you know will always result in mismatches\nthen you can ignore them so that they don't end up in your published results.\nMultiple ignore clauses can be stacked together.\n\n```elixir\ndef staff?(id) do\n  user = User.get(id)\n\n  experiment(\"staff?\")\n  |\u003e control(fn -\u003e old_role_check(user) end)\n  |\u003e candidate(fn -\u003e Roles.staff?(user) end)\n  |\u003e ignore(fn control, candidate -\u003e\n    # If the control passed but the candidate didn't check to see if its because\n    # the user has no email. We haven't implemented that in our new system yet.\n    control \u0026\u0026 !candidate \u0026\u0026 user.email == \"\"\n  end)\n  # Admin users are always considered staff.\n  |\u003e ignore(fn _, _, -\u003e Roles.admin?(user) end)\nend\n```\n\n## Exception handling\n\nAlchemy tries to be as transparent as possible. Because of this *all* errors\nraised in either the control or the candidate are rescued and stored in the result.\nThis allows you to see any errors in your candidates or control when you publish your results.\n\nOnce the results have been published the control value is returned. If the control\nraised an error during its execution that error *will be reraised* with its original stacktrace.\n\n## Execution details\n\nExecuting the control, candidates, and publishing are all done *sequentially*\nwithin the calling process. Trying to run these operations concurrently introduces\nmultiple failure conditions and possibility for timeouts. These failure scenarios\ntend to surprise users and breaks one of Alchemy's core goals of being transparent\nand making refactors safe. If you want to run any of these operations concurrently\nthen its best to do that in your own code so that you aren't surprised and can\nhandle failures in the way that makes the most sense for your application.\n\n## Prior Art\n\nPractically all of the concepts and many of the examples are shamelessly stolen\nfrom Github's amazing [Scientist](https://github.com/github/scientist) project.\nAll credit goes to them for popularizing this idea and creating such a smart api.\n\n## Contributing\n\nPull Requests and Issues are greatly appreciated!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-toniq%2Falchemy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felixir-toniq%2Falchemy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-toniq%2Falchemy/lists"}