{"id":13879393,"url":"https://github.com/salsify/avro-builder","last_synced_at":"2025-06-10T15:39:11.899Z","repository":{"id":45885962,"uuid":"54029712","full_name":"salsify/avro-builder","owner":"salsify","description":"Ruby DSL to create Avro schemas","archived":false,"fork":false,"pushed_at":"2025-01-03T19:06:23.000Z","size":173,"stargazers_count":111,"open_issues_count":9,"forks_count":7,"subscribers_count":55,"default_branch":"master","last_synced_at":"2025-05-17T07:19:05.083Z","etag":null,"topics":["avro","gem","hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salsify.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-16T12:25:07.000Z","updated_at":"2025-01-03T19:06:26.000Z","dependencies_parsed_at":"2024-11-24T08:31:42.693Z","dependency_job_id":"12ab528a-6efb-4a21-8bf5-4b60f4346965","html_url":"https://github.com/salsify/avro-builder","commit_stats":{"total_commits":49,"total_committers":9,"mean_commits":5.444444444444445,"dds":0.5510204081632653,"last_synced_commit":"99b41d9d59640593bc360a41c17727793e7f01c9"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-builder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-builder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-builder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-builder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salsify","download_url":"https://codeload.github.com/salsify/avro-builder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favro-builder/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259104094,"owners_count":22805808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","gem","hacktoberfest"],"created_at":"2024-08-06T08:02:19.451Z","updated_at":"2025-06-10T15:39:11.835Z","avatar_url":"https://github.com/salsify.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Avro::Builder\n\n[![Build Status](https://circleci.com/gh/salsify/avro-builder.svg?style=svg)][circleci]\n[![Gem Version](https://badge.fury.io/rb/avro-builder.svg)](https://badge.fury.io/rb/avro-builder)\n\n[circleci]: https://circleci.com/gh/salsify/avro-builder\n\n`Avro::Builder` provides a Ruby DSL to create [Apache Avro](https://avro.apache.org/docs/current/) Schemas.\n\nThis DSL was created because:\n* The [Avro IDL](https://avro.apache.org/docs/current/idl-language/) is not supported in Ruby.\n* The Avro IDL can only be used to define Protocols.\n* Schemas can be extracted as JSON from an IDL Protocol but support\n  for imports is still limited.\n\nAdditional background on why we developed `avro-builder` is provided\n[here](http://blog.salsify.com/engineering/adventures-in-avro).\n\n## Features\n* The syntax is designed for ease-of-use.\n* Definitions can be imported by name. This includes auto-loading from a configured\n  set of paths. This allows definitions to split across files and even reused\n  between projects.\n* Record definitions can inherit from other record definitions.\n* [Schema Store](#schema-store) to load files written in the DSL and return\n  `Avro::Schema` objects.\n\n## Limitations\n\n* Only Avro Schemas, not Protocols are supported.\n* See [Issues](https://github.com/salsify/avro-builder/issues) for functionality\n  that has yet to be implemented.\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'avro-builder'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install avro-builder\n\n## Railtie\n\nWhen included in a Rails project, `#{Rails.root}/avro/dsl` is configured as a\nload path for the DSL.\n\nA [rake task](#avro-generate-rake-task) is also defined for generating Avro JSON\nschemas from the DSL.\n\n## Usage\n\nTo use `Avro::Builder`, define a schema:\n\n```ruby\nnamespace 'com.example'\n\nfixed :password, 8\n\nenum :user_type, :ADMIN, :REGULAR\n\nrecord :user do\n  required :id, :long\n  required :user_name, :string\n  required :type, :user_type, default: :REGULAR\n  required :pw, :password\n  optional :full_name, :string\n  required :nicknames, :array, items: :string\n  required :permissions, :map, values: :bytes\nend\n```\n\nThe schema definition may be passed as a string or a block to\n`Avro::Builder.build`.\n\nThis generates the following Avro JSON schema:\n```json\n{\n  \"type\": \"record\",\n  \"name\": \"user\",\n  \"namespace\": \"com.example\",\n  \"fields\": [\n    {\n      \"name\": \"id\",\n      \"type\": \"long\"\n    },\n    {\n      \"name\": \"user_name\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"type\",\n      \"type\": {\n        \"name\": \"user_type\",\n        \"type\": \"enum\",\n        \"symbols\": [\n          \"ADMIN\",\n          \"REGULAR\"\n        ],\n        \"namespace\": \"com.example\"\n      },\n      \"default\": \"REGULAR\"\n    },\n    {\n      \"name\": \"pw\",\n      \"type\": {\n        \"name\": \"password\",\n        \"type\": \"fixed\",\n        \"size\": 8,\n        \"namespace\": \"com.example\"\n      }\n    },\n    {\n      \"name\": \"full_name\",\n      \"type\": [\n        \"null\",\n        \"string\"\n      ],\n      \"default\": null\n    },\n    {\n      \"name\": \"nicknames\",\n      \"type\": {\n        \"type\": \"array\",\n        \"items\": \"string\"\n      }\n    },\n    {\n      \"name\": \"permissions\",\n      \"type\": {\n        \"type\": \"map\",\n        \"values\": \"bytes\"\n      }\n    }\n  ]\n}\n```\n\n### Required and Optional\n\nFields for a record are specified as `required` or `optional`. Optional fields are\nimplemented as a union in Avro, where `null` is the first type in the union and\nthe field has a default value of `null`.\n\n### Named Types\n\n`fixed` and `enum` fields may be specified inline as part of a record\nor as standalone named types.\n\n```ruby\n# Either syntax is supported for specifying the size\nfixed :f, 4\nfixed :g, size: 8\n\n# Either syntax is supported for specifying symbols\nenum :e, :X, :Y, :Z\nenum :d, symbols: [:A, :B]\n\n# defaults can be set for enums with Ruby Avro v1.10.0\nenum :c, symbols: [:A, :B], default: :A\n\nrecord :my_record_with_named do\n  required :f_ref, :f\n  required :fixed_inline, :fixed, size: 9\n  required :e_ref, :e\n  required :enum_inline, :enum, symbols: [:P, :Q]\nend\n```\n\n### Complex Types\n\nArray, maps and unions can each be embedded within another complex type using\nmethods that match the type name:\n\n```ruby\nrecord :complex_types do\n  required :array_of_unions, :array, items: union(:int, :string)\n  required :array_or_map, :union, types: [array(:int), map(:int)]\nend\n```\n\nMethods may also be used for complex types instead of separately specifying the\ntype name and options:\n\n```ruby\nrecord :complex_types do\n  required :array_of_unions, array(union(:int, :string))\n  required :array_or_map, union(array(:int), map(:int))\nend\n```\n\nFor more on unions see [below](#unions).\n\n### Nested Records\n\nNested records may be created by referring to the name of the previously\ndefined record or using the field type `:record`.\n\n```ruby\nrecord :sub_rec do\n  required :i, :int\nend\n\nrecord :top_rec do\n  required :sub, :sub_rec\nend\n```\n\nDefinining a subrecord inline:\n\n```ruby\nrecord :my_rec do\n  required :nested, :record do\n    required :s, :string\n  end\nend\n```\n\nNested record types defined without an explicit name are given a generated\nname based on the name of the field and record that they are nested within.\nIn the example above, the nested record type would have the generated name\n`__my_rec_nested_record`:\n\n```json\n{\n  \"type\": \"record\",\n  \"name\": \"my_rec\",\n  \"fields\": [\n    {\n      \"name\": \"nested\",\n      \"type\": {\n        \"type\": \"record\",\n        \"name\": \"__my_rec_nested_record\",\n        \"fields\": [\n          {\n            \"name\": \"s\",\n            \"type\": \"string\"\n          }\n        ]\n      }\n    }\n  ]\n}\n```\n\n### Unions\n\nA union may be specified within a record using `required` and `optional` with\nthe `:union` type:\n\n```ruby\nrecord :my_record_with_unions do\n  required :req_union, :union, types: [:string, :int]\n  optional :opt_union, :union, types: [:float, :long]\nend\n```\n\nFor an optional union, `null` is automatically added as the first type for\nthe union and the field defaults to `null`.\n\nUnions may also be defined using the `union` method instead of specifying the\n`:union` type and member types separately:\n\n```ruby\nrecord :my_record_with_unions do\n  required :req_union, union(:string, :int)\n  optional :opt_union, union(:float, :long)\nend\n```\n\n### Logical Types\n\nThe DSL supports setting a logical type on any type except a union. The Avro\n[spec](https://avro.apache.org/docs/current/spec.html#Logical+Types) lists the logical types\nthat are currently defined. Note: `avro-builder` is more permissive and any logical type can\nbe specified on a type.\n\nA logical type can be specified for a field using the `logical_type` attribute:\n\n```ruby\nrecord :with_timestamp\n required :created_at, :long, logical_type: 'timestamp-micros'\nend\n```\n\nPrimitive types with a logical type can also be embedded within complex types\nusing either the generic `type` method:\n\n```ruby\nrecord :with_date_array\n  required :date_array, :array, type(:int, logical_type: date)\nend\n```\n\nOr using a primitive type specific method:\n\n```ruby\nrecord :with_date_array\n  required :date_array, :array, int(logical_type: date)\nend\n```\n\n#### Decimal Logical Types\n\nThe decimal logical type, for bytes and fixed types, is currently the only logical type that requires additional\nattributes. For decimals, precision must be specified and scale may optionally be specified. `avro-builder`\nsupports both of these attributes for bytes and fixed decimals. See the Avro\n[spec](https://avro.apache.org/docs/current/spec.html#Decimal) for more details.\n\n### Abstract Types\n\nTypes can be declared as abstract in the DSL. Declaring a type as abstract\nprevents the rake task from generating an Avro JSON schema for the type.\n\nA type can be declared as abstract using either an option or a method in the\nDSL when defining the type:\n\n```ruby\nrecord :unique_id, abstract: true\n  required :uuid, :fixed, size: 38\nend\n\nenum :status do\n  symbols %w(valid invalid)\n  abstract true\nend\n```\n\n### Type Macros\n\n`avro-builder` allows type macros to be defined that expand to types that\ncannot normally be named in Avro schemas. These macro names are not retained\nin generated schemas but allow definitions to be reused across DSL files:\n\n```ruby\ntype_macro :timestamp, long(logical_type: 'timestamp-millis')\n\nrecord :user do\n  required :created_at, :timestamp\n  required :updated_at, :timestamp\nend\n```\n\nType macros inherit the namespace from the context where they are defined or\nan explicit namespace option may be specified:\n\n```ruby\ntype_macro :timestamp, long(logical_type: 'timestamp-millis'),\n           namespace: 'com.my_company'\n```\n\nType macros are always marked as abstract and do not generate an Avro JSON\nschema file when using the rake task.\n\n### Auto-loading and Imports\n\nSpecify paths to search for definitions:\n\n```ruby\nAvro::Builder.add_load_path('/path/to/dsl/files')\n```\n\nUndefined references are automatically loaded from a file with the same name.\nThe load paths are searched for `.rb` file with a matching name.\n\nFiles may also be explicitly imported using `import \u003cfilename\u003e`.\n\n### Extends\n\nA previously defined record may be referenced in the definition of another\nrecord using `extends \u003crecord_name\u003e`. This adds all of the fields from\nthe referenced record to the current record. The current record may override\nfields in the record that it extends.\n\n```\nrecord :original do\n  required :first, :string\n  required :second, :int\nend\n\nrecord :extended do\n  extends :original\n  optional :first, :string\nend\n```\n\nAdditionally you can provide a `namespace` to `extends` if necessary to remove ambiguity.\n\n```\nnamespace 'com.newbie'\n\nrecord :original, namespace: 'com.og' do\n  required :first, :string\n  required :second, :int\nend\n\nrecord :original do\n  required :first, :string\n  required :second, :int\nend\n\nrecord :extended do\n  extends :original, namespace: 'com.og'\n  optional :first, :string\nend\n```\n\n\n## Schema Store\n\nThe `Avro::Builder::SchemaStore` can be used to load DSL files and return cached\n`Avro::Schema` objects. This schema store can be used as the schema store for\n[avromatic](https://github.com/salsify/avromatic)\nto generate models directly from schemas defined using the DSL.\n\nThe schema store must be initialized with the path where DSL files are located:\n\n```ruby\nschema_store = Avro::Builder::SchemaStore.new(path: '/path/to/dsl/files')\nschema_store.find('schema_name', 'my_namespace')\n#=\u003e Avro::Schema (for file at '/path/to/dsl/files/my_namespace/schema_name.rb')\n```\n\nTo configure `Avromatic` to use this schema store and its Messaging API:\n\n```ruby\nAvromatic.configure do |config|\n  config.schema_store = Avro::Builder::SchemaStore.new(path: 'avro/dsl')\n  config.registry_url = 'https://builder:avro@avro-schema-registry.salsify.com'\n  config.build_messaging!\nend\n```\n\n### Avro Generate Rake Task\n\nThere is a rake task that can be used to generate Avro schemas from all DSL\nfiles.\n\nA rake task is automatically defined via a Railtie for Rails projects that uses\n`#{Rails.root}/avro/dsl` as the root for Avro DSL files.\n\nCustom rake tasks can also be defined:\n\n```ruby\nrequire 'avro/builder/rake/avro_generate_task'\nAvro::Builder::Rake::AvroGenerateTask.new(name: :custom_gen,\n                                          dependencies: [:load_app]) do |task|\n  task.filetype = 'avsc' # default option\n  task.root = '/path/to/dsl/files'\n  task.load_paths \u003c\u003c '/additional/dsl/files'\nend\n```\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nIssues and pull requests are welcome on GitHub at https://github.com/salsify/avro-builder.\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalsify%2Favro-builder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalsify%2Favro-builder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalsify%2Favro-builder/lists"}