{"id":13491775,"url":"https://github.com/elixir-toniq/norm","last_synced_at":"2025-05-14T01:08:13.638Z","repository":{"id":41815589,"uuid":"177801621","full_name":"elixir-toniq/norm","owner":"elixir-toniq","description":"Data specification and generation","archived":false,"fork":false,"pushed_at":"2025-03-28T10:35:33.000Z","size":366,"stargazers_count":692,"open_issues_count":20,"forks_count":29,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-03T01:44:21.073Z","etag":null,"topics":["elixir","property-based-testing","specifcation"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elixir-toniq.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-03-26T14:10:50.000Z","updated_at":"2025-04-02T23:55:26.000Z","dependencies_parsed_at":"2023-01-21T20:46:19.842Z","dependency_job_id":"beda1097-84b6-41b4-8617-56f5989e28c0","html_url":"https://github.com/elixir-toniq/norm","commit_stats":{"total_commits":142,"total_committers":16,"mean_commits":8.875,"dds":0.4295774647887324,"last_synced_commit":"be1c31bc33ae10723b3d4fe8b9b3a2ffce90b710"},"previous_names":["keathley/norm"],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Fnorm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Fnorm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Fnorm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elixir-toniq%2Fnorm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elixir-toniq","download_url":"https://codeload.github.com/elixir-toniq/norm/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248154646,"owners_count":21056541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elixir","property-based-testing","specifcation"],"created_at":"2024-07-31T19:01:00.061Z","updated_at":"2025-04-10T03:40:08.946Z","avatar_url":"https://github.com/elixir-toniq.png","language":"Elixir","funding_links":[],"categories":["Elixir"],"sub_categories":[],"readme":"# Norm\n\n\u003c!-- MDOC !--\u003e\n\nNorm is a system for specifying the structure of data. It can be used for\nvalidation and for generation of data. Norm does not provide any set of\npredicates and instead allows you to re-use any of your existing\nvalidations.\n\n```elixir\nimport Norm\n\niex\u003e conform!(123, spec(is_integer() and \u0026(\u00261 \u003e 0)))\n123\n\niex\u003e conform!(-50, spec(is_integer() and \u0026(\u00261 \u003e 0)))\n** (Norm.MismatchError) Could not conform input:\nval: -50 fails: \u0026(\u00261 \u003e 0)\n\niex\u003e user_schema = schema(%{\n...\u003e   user: schema(%{\n...\u003e     name: spec(is_binary()),\n...\u003e     age: spec(is_integer() and \u0026(\u00261 \u003e 0))\n...\u003e   })\n...\u003e })\niex\u003e input = %{user: %{name: \"chris\", age: 30, email: \"c@keathley.io\"}}\niex\u003e conform!(input, user_schema)\n%{user: %{name: \"chris\", age: 30, email: \"c@keathley.io\"}}\niex\u003e generated_users =\n...\u003e   user_schema\n...\u003e   |\u003e gen()\n...\u003e   |\u003e Enum.take(3)\niex\u003e for g \u003c- generated_users, do: g.user.age \u003e 0 \u0026\u0026 is_binary(g.user.name)\n[true, true, true]\n```\n\nNorm can also be used to specify contracts for function definitions:\n\n```elixir\ndefmodule Colors do\n  use Norm\n\n  def rgb(), do: spec(is_integer() and \u0026(\u00261 in 0..255))\n\n  def hex(), do: spec(is_binary() and \u0026String.starts_with?(\u00261, \"#\"))\n\n  @contract rgb_to_hex(r :: rgb(), g :: rgb(), b :: rgb()) :: hex()\n  def rgb_to_hex(r, g, b) do\n    # ...\n  end\nend\n```\n\n## Validation and conforming values\n\nNorm validates data by \"conforming\" the value to a specification. If the\nvalues don't conform then a list of errors is returned. There are\n2 functions provided for this `conform/2` and `conform!/2`. If you need to\nreturn a list of well defined errors then you should use `conform/2`.\nOtherwise `conform!/2` is generally more useful. The input data is\nalways passed as the 1st argument to `conform` so that calls to conform\nare easily chainable.\n\n### Predicates and specs\n\nNorm does not provide a special set of predicates and instead allows you\nto convert any predicate into a spec with the `spec/1` macro. Predicates\ncan be composed together using the `and` and `or` keywords. You can also\nuse anonymous functions to create specs.\n\n```elixir\nspec(is_binary())\nspec(is_integer() and \u0026(\u00261 \u003e 0))\nspec(is_binary() and fn str -\u003e String.length(str) \u003e 0 end)\n```\n\nThe data is always passed as the first argument to your predicate so you\ncan use predicates with multiple values like so:\n\n```elixir\niex\u003e defmodule Predicate do\n...\u003e   def greater?(x, y), do: x \u003e y\n...\u003e end\niex\u003e conform!(10, spec(Predicate.greater?(5)))\n10\niex\u003e conform!(3, spec(Predicate.greater?(5)))\n** (Norm.MismatchError) Could not conform input:\nval: 3 fails: Predicate.greater?(5)\n```\n\n### Tuples and atoms\n\nAtoms and tuples can be matched without needing to wrap them in a function.\n\n```elixir\niex\u003e :atom = conform!(:atom, :atom)\n:atom\niex\u003e {1, \"hello\"} = conform!({1, \"hello\"}, {spec(is_integer()), spec(is_binary())})\n{1, \"hello\"}\niex\u003e conform!({1, 2}, {:one, :two})\n** (Norm.MismatchError) Could not conform input:\nval: 1 in: 0 fails: is not an atom.\nval: 2 in: 1 fails: is not an atom.\n```\n\nBecause Norm supports matching on bare tuples we can easily validate functions\nthat return `{:ok, term()}` and `{:error, term()}` tuples. These specifications can be combined with `one_of/1` to create union types.\n\n```elixir\niex\u003e defmodule User do\n...\u003e   defstruct [:name, :age]\n...\u003e\n...\u003e   def get_name(id) do\n...\u003e     case id do\n...\u003e       1 -\u003e {:ok, \"Chris\"}\n...\u003e       2 -\u003e {:ok, \"Alice\"}\n...\u003e       _ -\u003e {:error, \"user does not exist\"}\n...\u003e     end\n...\u003e   end\n...\u003e end\niex\u003e result_spec = one_of([\n...\u003e   {:ok, spec(is_binary())},\n...\u003e   {:error, spec(fn _ -\u003e true end)},\n...\u003e ])\niex\u003e {:ok, _name} = conform!(User.get_name(1), result_spec)\n{:ok, \"Chris\"}\niex\u003e {:ok, \"Alice\"} = conform!(User.get_name(2), result_spec)\n{:ok, \"Alice\"}\niex\u003e {:error, _error} = conform!(User.get_name(-42), result_spec)\n{:error, \"user does not exist\"}\n```\n\n### Collections\n\nNorm can define collections of values using `coll_of`.\n\n```elixir\niex\u003e conform!([1,2,3], coll_of(spec(is_integer)))\n[1, 2, 3]\n```\n\nCollections can take a number of options:\n\n* `:kind` - predicate function the kind of collection being conformed\n* `:distinct` - boolean value for specifying if the collection should have distinct elements\n* `:min_count` - Minimum element count\n* `:max_count` - Maximum element count\n* `:into` - The output collection the input will be conformed into. If not specified then the input type will be used.\n\n```elixir\niex\u003e conform!([:a, :b, :c], coll_of(spec(is_atom), into: MapSet.new()))\n#MapSet\u003c[:a, :b, :c]\u003e\n```\n\n### Schemas\n\nNorm provides a `schema/1` function for specifying maps and structs:\n\n```elixir\niex\u003e user_schema = schema(%{\n...\u003e   user: schema(%{\n...\u003e     name: spec(is_binary()),\n...\u003e     age: spec(is_integer() and \u0026 \u00261 \u003e 0),\n...\u003e   })\n...\u003e })\niex\u003e conform!(%{user: %{name: \"chris\", age: 31}}, user_schema)\n%{user: %{name: \"chris\", age: 31}}\niex\u003e conform!(%{user: %{name: \"chris\", age: -31}}, user_schema)\n** (Norm.MismatchError) Could not conform input:\nval: -31 in: :user/:age fails: \u0026(\u00261 \u003e 0)\n```\n\nSchema's are designed to allow systems to grow over time. They provide this\nfunctionality in two ways. The first is that any unspecified fields in the input\nare passed through when conforming the input. The second is that all keys in a\nschema are optional. This means that all of these are valid:\n\n```elixir\niex\u003e user_schema = schema(%{\n...\u003e   name: spec(is_binary()),\n...\u003e   age: spec(is_integer()),\n...\u003e })\niex\u003e conform!(%{}, user_schema)\n%{}\niex\u003e conform!(%{age: 31}, user_schema)\n%{age: 31}\niex\u003e conform!(%{foo: :foo, bar: :bar}, user_schema)\n%{foo: :foo, bar: :bar}\n```\n\nIf you're used to more restrictive systems for managing data these might seem\nlike odd choices. We'll see how to specify required keys when we discuss Selections.\n\n#### Structs\n\nYou can also create specs from structs:\n\n```elixir\ndefmodule User do\n  defstruct [:name, :age]\n\n  def s, do: schema(%__MODULE__{\n      name: spec(is_binary()),\n      age: spec(is_integer())\n    })\nend\n```\n\nThis will ensure that the input is a `User` struct with the key that match\nthe given specification. Its convention to provide a `s()` function in the\nmodule that defines the struct so that schema's can be shared throughout\nyour system.\n\nYou don't need to provide specs for all the keys in your struct. Only the\nspecced keys will be conformed. The remaining keys will be checked for\npresence.\n\n```elixir\ndefmodule Norm.User do\n  defstruct [:name, :age]\nend\niex\u003e user_schema = schema(%Norm.User{})\niex\u003e conform!(%Norm.User{name: \"chris\"}, user_schema)\n```\n\n#### Key semantics\n\nAtom and string keys are matched explicitly and there is no casting that\noccurs when conforming values. If you need to match on string keys you\nshould specify your schema with string keys.\n\nSchemas accomodate growth by disregarding any unspecified keys in the input map.\nThis allows callers to start sending new data over time without coordination\nwith the consuming function.\n\n### Selections and optionality\n\nWe said that all of the fields in a schema are optional. In order to specify\nthe keys that are required in a specific use case we can use a Selection. The\nSelections takes a schema and a list of keys - or keys to lists of keys - that\nmust be present in the schema.\n\n```elixir\niex\u003e user_schema = schema(%{\n...\u003e   user: schema(%{\n...\u003e     name: spec(is_binary()),\n...\u003e     age: spec(is_integer()),\n...\u003e   })\n...\u003e })\niex\u003e just_age = selection(user_schema, [user: [:age]])\niex\u003e conform!(%{user: %{name: \"chris\", age: 31}}, just_age)\n%{user: %{age: 31, name: \"chris\"}}\niex\u003e conform!(%{user: %{name: \"chris\"}}, just_age)\n** (Norm.MismatchError) Could not conform input:\nval: %{name: \"chris\"} in: :user/:age fails: :required\n```\n\nIf you need to mark all fields in a schema as required you can elide the list\nof keys like so:\n\n```elixir\niex\u003e user_schema = schema(%{\n...\u003e   user: schema(%{\n...\u003e     name: spec(is_binary()),\n...\u003e     age: spec(is_integer()),\n...\u003e   })\n...\u003e })\niex\u003e conform!(%{user: %{name: \"chris\", age: 31}}, selection(user_schema))\n%{user: %{name: \"chris\", age: 31}}\n```\n\nSelections are an important tool because they give control over optionality\nback to the call site. This allows callers to determine what they actually need\nand makes schema's much more reusable.\n\n### Patterns\n\nNorm provides a way to specify alternative specs using the `alt/1`\nfunction. This is useful when you need to support multiple schema's or\nmultiple alternative specs.\n\n```elixir\niex\u003e create_event = schema(%{type: spec(\u0026(\u00261 == :create))})\niex\u003e update_event = schema(%{type: spec(\u0026(\u00261 == :update))})\niex\u003e event = alt(create: create_event, update: update_event)\niex\u003e conform!(%{type: :create}, event)\n{:create, %{type: :create}}\niex\u003e conform!(%{type: :update}, event)\n{:update, %{type: :update}}\niex\u003e conform!(%{type: :delete}, event)\n** (Norm.MismatchError) Could not conform input:\nval: :delete in: :create/:type fails: \u0026(\u00261 == :create)\nval: :delete in: :update/:type fails: \u0026(\u00261 == :update)\n```\n\n## Generators\n\nAlong with validating that data conforms to a given specification, Norm\ncan also use specificiations to generate examples of good data. These\nexamples can then be used for property based testing, local development,\nseeding databases, or any other use case.\n\n```elixir\niex\u003e user_schema = schema(%{\n...\u003e   name: spec(is_binary()),\n...\u003e   age: spec(is_integer() and \u0026(\u00261 \u003e 0))\n...\u003e })\niex\u003e generated =\n...\u003e   user_schema\n...\u003e   |\u003e gen()\n...\u003e   |\u003e Enum.take(3)\niex\u003e for user \u003c- generated, do: user.age \u003e 0 \u0026\u0026 is_binary(user.name)\n[true, true, true]\n```\n\nUnder the hood Norm uses StreamData for its data generation. This means\nyou can use your specs in tests like so:\n\n```elixir\ninput_data = schema(%{\"user\" =\u003e schema(%{\"name\" =\u003e spec(is_binary())})})\n\nproperty \"users can update names\" do\n  check all input \u003c- gen(input_data) do\n    assert :ok == update_user(input)\n  end\nend\n```\n\n### Built in generators\n\nNorm will try to infer the generator to use from the predicate defined in\n`spec`. It looks specifically for the guard clauses used for primitive\ntypes in elixir. Not all of the built in guard clauses are supported yet.\nPRs are very welcome ;).\n\n### Guiding generators\n\nYou may have specs like `spec(fn x -\u003e rem(x, 2) == 0 end)` which check to\nsee that an integer is even or not. This generator expects integer values\nbut there's no way for Norm to determine this. If you try to create\na generator from this spec you'll get an error:\n\n```elixir\ngen(spec(fn x -\u003e rem(x, 2) == 0 end))\n** (Norm.GeneratorError) Unable to create a generator for: fn x -\u003e rem(x, 2) == 0 end\n    (norm) lib/norm.ex:76: Norm.gen/1\n```\n\nYou can guide Norm to the right generator by specifying a guard clause as\nthe first predicate in a spec. If Norm can find the right generator then\nit will use any other predicates as filters in the generator.\n\n```elixir\nEnum.take(gen(spec(is_integer() and fn x -\u003e rem(x, 2) == 0 end)), 5)\n[0, -2, 2, 0, 4]\n```\n\nBut its also possible to create filters that are too specific such as\nthis:\n\n```elixir\ngen(spec(is_binary() and \u0026(\u00261 =~ ~r/foobarbaz/)))\n```\n\nNorm can determine the generators to use however its incredibly unlikely\nthat Norm will be able to generate data that matches the filter. After 25\nconsecutive unseccessful attempts to generate a good value Norm (StreamData\nunder the hood) will return an error. In these scenarios we can create\na custom generator.\n\n### Overriding generators\n\nYou'll often need to guide your generators into the interesting parts of the\nstate space so that you can easily find bugs. That means you'll want to tweak\nand control your generators. Norm provides an escape hatch for creating your\nown generators with the `with_gen/2` function:\n\n```elixir\nage = spec(is_integer() and \u0026(\u00261 \u003e= 0))\nreasonable_ages = with_gen(age, StreamData.integer(0..105))\n```\n\nBecause `gen/1` returns a StreamData generator you can compose your generators\nwith other StreamData functions:\n\n```elixir\nage = spec(is_integer() and \u0026(\u00261 \u003e= 0))\nStreamData.frequency([\n  {3, gen(age)},\n  {1, StreamData.binary()},\n])\n\ngen(age) |\u003e StreamData.map(\u0026Integer.to_string/1) |\u003e Enum.take(5)\n[\"1\", \"1\", \"3\", \"4\", \"1\"]\n```\n\nThis allows you to compose generators however you need to while keeping your\ngeneration co-located with the specification of the data.\n\n## Adding contracts to functions\n\nYou can `conform` data wherever it makes sense to do so in your application.\nBut one of the most common ways to use Norm is to validate a functions arguments\nand return value. Because this is such a common pattern, Norm provides function\nannotations similar to `@spec`:\n\n```elixir\ndefmodule Colors do\n  use Norm\n\n  def rgb(), do: spec(is_integer() and \u0026(\u00261 in 0..255))\n\n  def hex(), do: spec(is_binary() and \u0026String.starts_with?(\u00261, \"#\"))\n\n  @contract rgb_to_hex(r :: rgb(), g :: rgb(), b :: rgb()) :: hex()\n  @doc \"Convert an RGB value to its CSS-style hexadecimal notation.\"\n  def rgb_to_hex(r, g, b) do\n    # ...\n  end\nend\n```\n\nIf the arguments for `rgb_to_hex` don't conform to the specification or if\n`rgb_to_hex` does not return a value that conforms to `hex` then an error will\nbe raised.\n\nNote `@contract` must be placed _before_ `@doc` as above for ExDoc and\n`ExUnit.DocTest` to continue working as intended.\n\n\u003c!-- MDOC !--\u003e\n\n## Installation\n\nAdd `norm` to your list of dependencies in `mix.exs`. If you'd like to use\nNorm's generator capabilities then you'll also need to include StreamData\nas a dependency.\n\n```elixir\ndef deps do\n  [\n    {:stream_data, \"~\u003e 0.4\"},\n    {:norm, \"~\u003e 0.13\"}\n  ]\nend\n```\n\n## Should I use this?\n\nNorm is still early in its life so there may be some rough edges. But\nwe're actively using this at my current company (Bleacher Report) and\nworking to make improvements.\n\n## Contributing and TODOS\n\nNorm is being actively worked on. Any contributions are very welcome. Here is a\nlimited set of ideas that are coming soon.\n\n- [ ] More streamlined specification of keyword lists.\n- [ ] Support \"sets\" of literal values\n- [ ] specs for functions and anonymous functions\n- [ ] easier way to do dispatch based on schema keys\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-toniq%2Fnorm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felixir-toniq%2Fnorm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felixir-toniq%2Fnorm/lists"}