{"id":32167849,"url":"https://github.com/primait/avrogen","last_synced_at":"2026-02-23T15:01:09.753Z","repository":{"id":259207638,"uuid":"852653801","full_name":"primait/avrogen","owner":"primait","description":"Generate Elixir typedstructs and various useful helper functions from AVRO schemas at compile time.","archived":false,"fork":false,"pushed_at":"2026-02-17T16:53:42.000Z","size":237,"stargazers_count":3,"open_issues_count":13,"forks_count":0,"subscribers_count":27,"default_branch":"master","last_synced_at":"2026-02-17T17:57:43.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/primait.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-09-05T07:24:10.000Z","updated_at":"2026-02-02T10:55:09.000Z","dependencies_parsed_at":"2025-05-06T14:31:17.607Z","dependency_job_id":"626b4094-a3e6-47af-b54d-88db120d30da","html_url":"https://github.com/primait/avrogen","commit_stats":null,"previous_names":["primait/avrogen"],"tags_count":50,"template":false,"template_full_name":null,"purl":"pkg:github/primait/avrogen","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/primait%2Favrogen","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/primait%2Favrogen/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/primait%2Favrogen/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/primait%2Favrogen/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/primait","download_url":"https://codeload.github.com/primait/avrogen/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/primait%2Favrogen/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29746499,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T07:44:07.782Z","status":"ssl_error","status_checked_at":"2026-02-23T07:44:07.432Z","response_time":90,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-21T15:39:01.175Z","updated_at":"2026-02-23T15:01:09.747Z","avatar_url":"https://github.com/primait.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Avrogen\n\n[![Build Status](https://github.com/primait/avrogen/actions/workflows/elixir.yml/badge.svg)](https://github.com/primait/avrogen/actions/workflows/elixir.yml)\n[![Hex.pm](https://img.shields.io/badge/hex.pm-green)](https://hex.pm/packages/prima/avrogen)\n[![Documentation](https://img.shields.io/badge/documentation-gray)](https://prima.hexdocs.pm/avrogen/)\n\nGenerate Elixir typedstructs and various useful helper functions from AVRO\nschemas at compile time.\n\n## Rationale\n\nWhile there exists a handful of libraries to encode and decode AVRO messages in\nElixir, all of them consume schemas at runtime, which has the advantage of\nflexibilty e.g. this approach can be used with a schema registry, but you lose\nthe any compile time type safety for your types.\n\nAvrogen generates Elixir code from AVRO schemas, turning each record into module\ncontaining a `typedstruct` and a bunch of helper functions to encode and decode\nthe struct to and from AVRO binary format.\n\nFor example, the following schema...\n\n```json\n{\n  \"type\": \"record\",\n  \"namespace\": \"foo\",\n  \"name\": \"Bar\",\n  \"fields\": [\n    { \"name\": \"baz\", \"type\": [\"null\", \"string\"] },\n    { \"name\": \"qux\", \"type\": \"int\" }\n  ]\n}\n```\n\n... generates a module `foo/Bar.ex` which (with documentation and various bits\nof implementation omitted for the sake of brevity) looks like this:\n\n```elixir\ndefmodule Avro.Foo.Bar do\n  use TypedStruct\n  \n  @expected_keys MapSet.new([\"baz\", \"qux\"])\n  @pii_fields MapSet.new([])\n\n  typedstruct do\n    field :baz, nil | String.t()\n    field :qux, integer(), enforce: true\n  end\n\n  def avro_fqn(), do: \"foo.Bar\"\n  def to_avro_map(...) do ... end\n  def from_avro_map(...) do ... end\n  def pii_fields(), do: @pii_fields\n  def drop_pii(...) do ... end\n  def random_instance(rand_state) do ... end\nend\n```\n\nThe main feature here is the `typedstruct`, which allows us to initialize this\nmodule using the struct syntax:\n\n```elixir\n%Avro.Foo.Bar{\n  baz: \"quux\",\n  qux: 12\n}\n```\n\nThe other helper functions provide extra functionality which are explained\nbelow.\n\n## Installation\n\nIf [available in Hex](https://hex.pm/docs/publish), the package can be installed\nby adding `avrogen` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:avrogen, \"~\u003e 0.4.3\", organization: \"prima\"}\n  ]\nend\n```\n\n## Usage\n\nThe easiest way to use avrogen is to add `:avro_code_generator` to your list of\ncompilers in your `mix.exs` file, making sure to place it before the other\ncompilers so all Elixir code is in place before the Elixir compiler runs.\n\n```elixir\ncompilers: [:avro_code_generator | Mix.compilers()]\n```\n\nYou'll also need to tell the elixir compiler to build the generated code, which\ncan be acheived by adding the `generated` dir (the default destination\ndirectory) to your `elixirc_paths`.\n\n```elixir\nelixirc_paths: [\"lib\", \"generated\"]\n```\n\nNow, you can create a new directory called `schemas` at the root of your project\nand put some `.avsc` files in there. They will be built and compiled whenever\nthings need to get recompiled, so just run your mix commands as usual.\n\n### Options\n\nWhile the defaults might be OK for some folks, you can configure the generator\ntask from your mix.exs file, using the `avro_code_generator_opts` key.\n\nE.g.\n\n```elixir\navro_code_generator_opts: [\n  paths: [\"schemas/*.avsc\"],\n  dest: \"generated\",\n  schema_root: \"schema\",\n  module_prefix: \"Avro\",\n  scoped_embed_paths: [\"priv/schema/events.*.avsc\"],\n  schema_resolution_mode: :flat\n]\n```\n\nThe options are:\n\n- `paths` - an array of file paths or wildcards to locate schema files. Defaults\n  to `\"schemas/*.avsc\"`.\n- `dest` - A directory of where to put the generated elixir code. Defaults to\n  `\"generated\"`.\n- `schema_root` - The root of the schema directory, this is the root dir that\n  will be used to resolve schemas located in other files. Defaults to\n  `\"schemas\"`\n- `module_prefix` - String to place at the front of generated elixir modules.\n  Defaults to `\"Avro\"`.\n- `schema_resolution_mode` - Tells the code generator how to resolve external\n  schemas to a filename. Defaults to `:flat`.\n- `scope_embed_paths` - the glob patterns of the files where any embedded scopes\n  should have the generated module path contain the encompasing types.\n\n  For example, for the following schema\n\n  ```json\n  {\n    \"name\": \"Event\",\n    \"namespace\": \"events\",\n    \"type\": \"record\",\n    \"fields\": [\n      {\n        \"name\": \"details\",\n        \"type\": {\n          \"name\": \"Subtype\",\n          \"type\": \"record\",\n          \"fields\": [\n            ...\n          ]\n        }\n      }\n    ]\n  }\n  ```\n\n  If this file is included in the scoped_embed_paths, then the generated module\n  for `Subtype` would be called `Events.Event.Subtype` otherwise it would be\n  `Events.Subtype`. This option is useful when you have naming clashes in\n  embedded schema subtypes, or if you simply want to namespace subtypes to avoid\n  potential future clashes\n\n### Using the generated code\n\nFirstly, you'll need to start the Schema Registry process by adding the\nfollowing entry to your Application file:\n\n```elixir\n@impl true\ndef start(_type, _args) do\n  children = [\n    ...\n    # Start a schema registry\n    {Avrogen.Schema.SchemaRegistry, Application.get_application(__MODULE__)},\n    ...\n  ]\n  ...\nend\n```\n\nNow you can create new records in code using the full module name, which is\ncomprised of your prefix + the namespace + name of the record.\n\nE.g. the record:\n\n```json\n{\n  \"namespace\": \"foo\",\n  \"name\": \"Bar\",\n  \"fields\": [\n    {\"name\": \"quz\", \"type\": ...}\n  ]\n}\n```\n\n... will result in a module called `Avro.Foo.Bar`, which can be used like any\nother normal struct:\n\n```elixir\nmessage = %Avro.Foo.Bar{quz: ...}\n```\n\nEncode this module to a binary using the `Avrogen.encode_schemaless/1` function:\n\n```elixir\n{:ok, bytes} = Avrogen.encode_schemaless(message)\n```\n\nYou can decode it back into a struct using the `Avrogen.decode_schemaless/2`\nfunction, mind that you'll need to pass in the module name as avro binaries\ndon't encode their type.\n\n```elixir\n{:ok, message} = Avrogen.decode_schemaless(Avro.Foo.Bar, bytes)\n```\n\n### Converting to JSON\n\nEach generated module comes with the `@derive Jason.Encoder` attribute, which\ntells JSON that the struct can be encoded by simply serializing everything other\nthan the `__struct__` field. See https://hexdocs.pm/jason/Jason.Encoder.html for\nmore details.\n\nThus, converting a message to JSON is as simple as:\n\n```elixir\niex\u003e message = %Avro.Test.Person{name: \"Dave\", age: 37}\niex\u003e Jason.encode(message)\n{:ok, \"{\\\"age\\\":37,\\\"name\\\":\\\"Dave\\\"}\"}\n```\n\n\u003e Note: This is not the same as the official AVRO JSON encoding spec, and is\n\u003e mainly used for debugging / making messages human readable.\n\n### External Schema Resolution\n\nSchemas commonly depend on other schemas, which can be located in a different\nfile.\n\nThe code generator has two different modes for file resolution: tree mode and\nflat mode.\n\nBoth modes work from a root directory passed in via the `schema_root` code\ngenerator option (see above).\n\nIn `:flat` mode, schemas are expected to be a flat list of files in the root dir\nlike so:\n\n```\nname.space.SchemaName -\u003e root/name.space.SchemaName.avsc\n```\n\nThis is how libraries like python's `fastavro` expect schemas to be laid out.\n\nIn `:tree` mode, the namespace is split into directories like so:\n\n```\nname.space.SchemaName -\u003e root/name/space/SchemaName.avsc\n```\n\nThis is how libraries like `avrora` expect schemas to be laid out.\n\n### Types\n\nThe table below lists supported primitive AVRO types, and their corresponding\nElixir type:\n\n| AVRO Type | Elixir Type                    |\n| --------- | ------------------------------ |\n| `null`    | `nil`                          |\n| `int`     | `integer`                      |\n| `double`  | `float`                        |\n| `string`  | `String.t()`                   |\n| `bytes`   | `binary`                       |\n| `array`   | `list`                         |\n| `map`     | `%{ String.t() =\u003e Value.t() }` |\n\nThe following logical types are also supported:\n\n| AVRO Logical Type        | AVRO Underlying Type | Elixir Type     |\n| ------------------------ | -------------------- | --------------- |\n| `uuid`                   | `string`             | `String`        |\n| `decimal`                | `string`             | `Decimal`       |\n| `decimal`                | `bytes`              | `Decimal`       |\n| `big-decimal`            | `bytes`              | `Decimal`       |\n| `date`                   | `int`                | `Date`          |\n| `date`                   | `string`             | `Date`          |\n| `iso_date`               | `string`             | `Date`          |\n| `datetime`               | `string`             | `DateTime`      |\n| `iso_datetime`           | `string`             | `DateTime`      |\n| `time-millis`            | `int`                | `Time`          |\n| `time-micros`            | `long`               | `Time`          |\n| `timestamp-millis`       | `long`               | `DateTime`      |\n| `timestamp-micros`       | `long`               | `DateTime`      |\n| `local-timestamp-millis` | `long`               | `NaiveDateTime` |\n| `local-timestamp-micros` | `long`               | `NaiveDateTime` |\n\nThe following AVRO types are not supported (yet):\n\n- `float` (use `double`)\n- `long` (use `int`)\n- `fixed`\n\n### PII Fields\n\nAvrogen introduces an unofficial extension to AVRO schema specification which\ncan be used to mark record's fields as PII (Personally Identifiable\nInformation). Each generated record module gets a `drop_pii/1` function which\nrecursively strips away all fields marked as PII in the record, and any records\ncontained within.\n\nMark a field as PII by adding `pii: true` option to the field. For example\nimagine you are storing names and ages of people, and the name is PII (but the\nage isn't).\n\n```json\n{\n  \"type\": \"record\",\n  \"name\": \"Person\",\n  \"namespace\": \"example\",\n  \"fields\": [\n    { \"name\": \"name\", \"type\": [\"null\", \"string\"], \"pii\": true },\n    { \"name\": \"age\", \"type\": \"int\" }\n  ]\n}\n```\n\nThen you can simply call `drop_pii/1` on your record to replace all the PII\nfields with `nil` like so:\n\n```elixir\nex\u003e person = %Avro.Example.Person{name: \"John Smith\", age: 38} \n%Avro.Example.Person{age: 38, name: \"John Smith\"}\n\nex\u003e Avro.Example.Person.drop_pii(person)\n%Avro.Example.Person{age: 38, name: nil}\n```\n\n\u003e Note: Fields marked as PII must be of a union type containing a null.\n\nThe AVRO spec specifies that any extra fields in schemas are ignored, so schemas\ncontaining this extension are backwards compatible with other AVRO parsers, as\nthey will just ignore this field.\n\n### Random Instance Generators\n\nEach generated module contains a function to create a random instance of the\nrecord/enum. This can be useful for fuzz testing, among other things.\n\nE.g. Using the `Person` example above, the generated module will contain the\nfollowing function:\n\n```elixir\ndef random_instance(rand_state) do\n  # ...\nend\n```\n\n\u003e The function expects to be given an erlang random state type object, which can\n\u003e be seeded in one of many ways depending on what you want to do with it. The\n\u003e simplest way to create this random state is to generate it with the default\n\u003e generator - `:rand.seed(:default)`, as demonstrated below.\n\nYou can use this `random_instance/1` function to generate random instances of\nthe module's struct, for example:\n\n```elixir\niex\u003e state = :rand.seed(:default)\niex\u003e {state, person} = Avro.Example.Person.random_instance(state)\niex\u003e person\n%Avro.Foo.Bar{\n  name: \u003c\u003c29, 120, 54, 75, 84, 54, 70, 29, 48, 68, 87, 87\u003e\u003e,\n  age: 1812334491\n}\n```\n\nIn this example, `person` is a random instance of the Avro.Example.Person\nrecord, and `state` is the mutated state which can be used again to pass to the\nnext call to `random_instance/1`.\n\nThe various types produce random values according to the following rules:\n\n| Type                                          | Rule                                                                                                                |\n| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |\n| `null`                                        | Always produces `nil`.                                                                                              |\n| `union`                                       | Random instance of any of the types within the union, where each type is equally likely to be chosen.               |\n| `string`                                      | Random utf8 binary of up to 1000 codepoints, where each codepoint lies in the range `0 \u003c= codepoint \u003c 10,000`.      |\n| `int`, `double`, `big-decimal` (logical type) | Random value in the range `-2,147,483,648 \u003c= value \u003c 2,147,483,648`.                                                |\n| `iso_date` and `iso_datetime` (logical types) | Random value in the range `1970-01-01T00:00:00 \u003c= value \u003c 2045-01-01T00:00:00`.                                     |\n| `enum`                                        | One of the symbols selected at random, with each symbol having an equal probability of showing up.                  |\n| `array`                                       | Random list of up to 10 elements, where the value of each element is a random instance of the array's element type. |\n| `record`                                      | Random instance of that record, where each field of the record is generated randomly according to the above rules.  |\n\nYou can control how random you really want these random instances to be using\nsome more unofficial extensions to the avro spec. For example, you can specify\nthe max and min values of int type fields using the \"range\" specifier like so:\n\n```json\n{\n  \"name\": \"age\",\n  \"type\": \"int\",\n  \"range\": {\n    \"int\": {\n      \"max\": 80,\n      \"min\": 16\n    }\n  }\n}\n```\n\nNow when you call `random_instance/1`, the age field will be limited to the\nrange `16 \u003c= age \u003c 80`.\n\nStrings can be formatted according to semantic formatting. Currently the only\nsupported type is \"postcode\", but support for more types may well be added in\nthe future.\n\nE.g.\n\n```json\n{\n  \"name\": \"postcode\",\n  \"type\": \"string\",\n  \"range\": {\n    \"string\": {\n      \"semantic_type\": \"postcode\"\n    }\n  }\n}\n```\n\nNow, the postcode field will be limited to random postcodes (e.g. `BS23 7SX`),\nrather than completely random strings.\n\n## Schema Generation\n\nAVRO's avsc format is not always the easiest format to maintain. Because it uses\nJSON, variables and comments are not allowed. Thus, `avrogen` comes with a\nschema generator tool to help ease the process.\n\nThis tool can be optionally used by adding it to the list of compilers for your\nproject's `mix.exs` file like so:\n\n```elixir\ncompilers: [:avro_schema_generator | Mix.compilers()]\n```\n\nNote: If you are also using the `avro_code_generator`, then you will need to put\nthe schema generator before the code generator, as the code generator requires\nschemas in order to do its job.\n\nThen, also in your `mix.exs` file, configure the code generator using the\nfollowing options:\n\n```elixir\navro_schema_generator_opts: [\n  paths: [\"exs_schemas/**/*.exs\"],\n  dest: \"schemas\",\n  schema_resolution_mode: :flat\n],\n```\n\nWhere the options are as follows:\n\n- `paths`: A wildcard expression which matches the location of your schema\n  definition files. Defaults to `exs_schemas/**/*.exs`.\n- `dest`: Where to put generated schema files. Defaults to `schemas`.\n- `schema_resolution_mode`: How to structure the dest dir (see the schema\n  resolution section above). Options are `:flat` or `:tree`. Defaults to\n  `:flat`.\n\nSo what goes in these schema definition files? All files should contain a single\nmodule which implements the `Avrogen.Schema.SchemaModule` behaviour. For\nexample:\n\n```elixir\ndefmodule Person do\n  alias Avrogen.Schema.SchemaModule\n  @behaviour SchemaModule\n  @impl SchemaModule\n  def schema_name(), do: \"application_data.v2\"\n\n  @impl SchemaModule\n  def avsc(), do: avro_schema()\n\n  # \"type\": \"record\",\n  # \"name\": \"Person\",\n  # \"namespace\": \"example\",\n  # \"fields\": [\n  #   {\"name\": \"name\", \"type\": [\"null\", \"string\"], \"pii\": true},\n  #   {\"name\": \"age\", \"type\": \"int\"}\n  # ]\n\n  @person %{\n    type: :record,\n    name: \"Person \",\n    namespace: \"example\",\n    doc: \"Describes a person.\",\n    fields: [\n      %{\n        name: :name,\n        type: [:null, :string],\n        doc: \"\"\"\n        The name of the person.\n        \"\"\"\n      },\n      %{\n        name: :age,\n        type: :int,\n        doc: \"\"\"\n        The age of the person.\n        \"\"\"\n      }\n    ]\n  }\n\n  @avro_spec [\n    @person\n  ]\n\n  def avro_schema() do\n    Jason.encode!(@avro_spec)\n  end\n\n  @impl SchemaModule\n  def avro_schema_elixir() do\n    @avro_spec\n  end\nend\n```\n\nWhen you next compile the code with e.g. `mix compile`, the following avsc\nschema will be generated:\n\n```json\n{\n  \"doc\": \"Describes a person.\",\n  \"fields\": [\n    {\n      \"doc\": \"The name of the person.\\n\",\n      \"name\": \"name\",\n      \"type\": [\n        \"null\",\n        \"string\"\n      ]\n    },\n    {\n      \"doc\": \"The age of the person.\\n\",\n      \"name\": \"age\",\n      \"type\": \"int\"\n    }\n  ],\n  \"name\": \"Person \",\n  \"namespace\": \"example\",\n  \"type\": \"record\"\n}\n```\n\nThere's not much magic here, but it should be evident how elixir variables and\nconstructs can be used to reduce repetition in the definitions of the schemas.\nIt's worth noting that these schema definitions are used to define lists of\nschemas in one go. Each individual schema is pulled out and placed into the\ntarget destination, and the file name is structured like so:\n`\u003cdest\u003e/\u003cnamespace\u003e.\u003cname\u003e.avsc`, which is the appropriate format for the avro\ncode generator to use later down the line.\n\nNote that once you enable this tool, it completely takes over the `dest`\ndirectory, so any other files found in here will most likely be removed.\n\n## Using with Avrora\n\nAvrora is an Elixir library for encoding/decoding avro messages, with options to\nintegrate with a schema registry.\n\nAvrora can work in conjunction with avrogen quite nicely, with avrogen\ngenerating the elixir code, and avrora handling communication with the schema\nregistry and encoding/decoding of messages.\n\nAvrogen expects schemas to be stored in the filesystem in a \"tree\" style format,\nso make sure to set the option `schema_resolution_mode` to `:tree` for both\ngenerators. Once you have configured the avrora cache (see docs on their\n[README](https://hexdocs.pm/avrora/readme.html#usage)), you can then use\navrogen's typedstructs to create the messages and do some basic type/key\nchecking, and avrora to encode/decode the messages.\n\nFor example, to encode...\n\n```elixir\n%module{} = message = %Avro.Test.Person{name: \"John Smith\", age: 38}\nname = module.avro_fqn()\nmap = module.to_avro_map(message)\n{:ok, bytes} = Avrora.encode(map, schema_name: name)\n# Do something with the bytes\n```\n\n... and then to decode ...\n\n```elixir\n{:ok, [decoded]} = Avrora.decode(bytes, schema_name: Avro.Test.Person.avro_fqn())\n{:ok, message} = Avro.Test.Person.from_avro_map(decoded)\n# Do something with the message\n```\n\n\u003e Note: Of course, this assumes you know the decoded message type ahead of time.\n\u003e You can ask Avrora to infer the message type using magic headers by calling\n\u003e `decode/1` rather than `decode/2` (omitting the `schema_name` option), but it\n\u003e doesn't disclose the inferred schema name to the caller, which is not\n\u003e particularly useful.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprimait%2Favrogen","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprimait%2Favrogen","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprimait%2Favrogen/lists"}