{"id":13879939,"url":"https://github.com/salsify/avromatic","last_synced_at":"2025-06-10T15:38:53.891Z","repository":{"id":44573100,"uuid":"58155824","full_name":"salsify/avromatic","owner":"salsify","description":"Generate Ruby models from Avro schemas","archived":false,"fork":false,"pushed_at":"2025-03-24T20:02:53.000Z","size":529,"stargazers_count":90,"open_issues_count":6,"forks_count":16,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-05-29T09:12:38.331Z","etag":null,"topics":["avro","gem","hacktoberfest"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/salsify.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-05T19:36:32.000Z","updated_at":"2025-03-24T19:59:52.000Z","dependencies_parsed_at":"2024-06-19T05:29:28.364Z","dependency_job_id":"43e1f138-9273-4be2-bcac-8af95ba2d928","html_url":"https://github.com/salsify/avromatic","commit_stats":{"total_commits":137,"total_committers":18,"mean_commits":7.611111111111111,"dds":0.5985401459854014,"last_synced_commit":"95fb7db28ef27704142beedb6e9ba54af34c4d74"},"previous_names":[],"tags_count":81,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favromatic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favromatic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favromatic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favromatic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/salsify","download_url":"https://codeload.github.com/salsify/avromatic/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/salsify%2Favromatic/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259104048,"owners_count":22805798,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","gem","hacktoberfest"],"created_at":"2024-08-06T08:02:39.915Z","updated_at":"2025-06-10T15:38:53.868Z","avatar_url":"https://github.com/salsify.png","language":"Ruby","funding_links":[],"categories":["Ruby"],"sub_categories":[],"readme":"# Avromatic\n\n[![Build Status](https://circleci.com/gh/salsify/avromatic.svg?style=svg)][circleci]\n[![Gem Version](https://badge.fury.io/rb/avromatic.svg)](https://badge.fury.io/rb/avromatic)\n\n[circleci]: https://circleci.com/gh/salsify/avromatic\n\n`Avromatic` generates Ruby models from [Avro](http://avro.apache.org/) schemas\nand provides utilities to encode and decode them.\n\n**This README reflects Avromatic 2.0. Please see the\n[1-0-stable](https://github.com/salsify/avromatic/blob/1-0-stable/README.md) branch for Avromatic 1.0.**\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'avromatic'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install avromatic\n\nSee the [Logical Types](#logical-types) section below for details on using\nAvromatic with unreleased Avro features.\n\n## Usage\n\n### Configuration\n\n`Avromatic` supports the following configuration:\n\n#### Model Generation\n\n* **schema_store**: A schema store is required to load Avro schemas from the filesystem.\n  It should be an object that responds to `find(name, namespace = nil)` and\n  returns an `Avro::Schema` object. An `AvroTurf::SchemaStore` can be used.\n  The `schema_store` is unnecessary if models are generated directly from\n  `Avro::Schema` objects. See [Models](#models).\n* **nested_models**: An optional [ModelRegistry](https://github.com/salsify/avromatic/blob/master/lib/avromatic/model_registry.rb)\n  that is used to store, by full schema name, the generated models that are\n  embedded within top-level models. By default a new `Avromatic::ModelRegistry`\n  is created.\n* **eager_load_models**: An optional array of models, or strings with class\n  names for models, that are added to `nested_models` at the end of\n  `Avromatic.configure` and during code reloading in Rails applications. This\n  option is useful for defining models that will be extended when the load order\n  is important.\n* **allow_unknown_attributes**: Optionally allow model constructors to silently\n  ignore unknown attributes. Defaults to `false`. WARNING: Setting this to `true`\n  will result in incorrect union member coercions if an earlier union member is\n  satisfied by a subset of the latter union member's attributes.\n\n#### Custom Types\n\nSee the section below on configuring [Custom Types](#custom-type-configuration).\n\n#### Using a Schema Registry/Messaging API\n\nThe configuration options below are required when using a schema registry\n(see [Confluent Schema Registry](http://docs.confluent.io/2.0.1/schema-registry/docs/intro.html))\nand the [Messaging API](#messaging-api).\n\n* **schema_registry**: An `AvroSchemaRegistry::Client` or\n  `AvroTurf::ConfluentSchemaRegistry` object used to store Avro schemas\n  so that they can be referenced by id. Either `schema_registry` or\n  `registry_url` must be configured. If using `build_schema_registry!`, only\n  `registry_url` is required. See example below.\n* **registry_url**: URL for the schema registry. This must be configured when using\n  `build_schema_registry!`. The `build_schema_registry!` method may\n  be used to create a caching schema registry client instance based on other\n  configuration values.\n* **use_schema_fingerprint_lookup**: Avromatic supports a Schema Registry\n  [extension](https://github.com/salsify/avro-schema-registry#extensions) that\n  provides an endpoint to lookup existing schema ids by fingerprint.\n  A successful response from this GET request can be cached indefinitely.\n  The use of this additional endpoint can be disabled by setting this option to\n  `false` and this is recommended if using a Schema Registry that does not support\n  the endpoint.\n* **messaging**: An `AvroTurf::Messaging` object to be shared by all generated models\n  The `build_messaging!` method may be used to create a `Avromatic::Messaging`\n  instance based on the other configuration values.\n* **logger**: The logger to use for the schema registry client.\n\nExample using a schema registry:\n\n```ruby\nAvromatic.configure do |config|\n  config.schema_store = AvroTurf::SchemaStore.new(path: 'avro/schemas')\n  config.registry_url = Rails.configuration.x.avro_schema_registry_url\n\n  config.build_messaging!\nend\n```\n\nNOTE: `build_messaging!` ultimately calls `build_schema_registry!` so you don't have\nto call both.\n\n#### Decoding\n\n* **use_custom_datum_reader**: `Avromatic` includes a modified subclass of\n  `Avro::IO::DatumReader`. This subclass returns additional information about\n  the index of union members when decoding Avro messages. This information is\n  used to optimize model creation when decoding. By default this information\n  is included in the hash returned by the `DatumReader` but can be omitted by\n  setting this option to `false`.\n\n#### Encoding\n* **use_custom_datum_writer**: `Avromatic` includes a modified subclass of\n  `Avro::IO::DatumWriter`. This subclass supports caching avro encodings for\n  immutable models and uses additional information about the index of union\n  members to optimize the encoding of Avro messages. By default this\n  information is included in the hash passed to the encoder but can be omitted\n  by setting this option to `false`.\n\n\n### Models\n\nModels are defined based on an Avro schema for a record.\n\nThe Avro schema can be specified by name and loaded using the schema store:\n\n```ruby\nclass MyModel\n  include Avromatic::Model.build(schema_name: :my_model)\nend\n\n# Construct instances by passing in a hash of attributes\ninstance = MyModel.new(id: 123, name: 'Tesla Model 3', enabled: true)\n\n# Access attribute values with readers\ninstance.name # =\u003e \"Tesla Model 3\"\n\n# Models are immutable by default\ninstance.name = 'Tesla Model X' # =\u003e NoMethodError (private method `name=' called for #\u003cMyModel:0x00007ff711e64e60\u003e)\n\n# Booleans can also be accessed by '?' readers that coerce nil to false\ninstance.enabled? # =\u003e true\n\n# Models implement ===, eql? and hash\ninstance == MyModel.new(id: 123, name: 'Tesla Model 3', enabled: true) # =\u003e true\ninstance.eql?(MyModel.new(id: 123, name: 'Tesla Model 3', enabled: true)) # =\u003e true\ninstance.hash # =\u003e -1279155042741869898\n\n# Retrieve a hash of the model's attributes via to_h, to_hash or attributes\ninstance.to_h # =\u003e {:id=\u003e123, :name=\u003e\"Tesla Model 3\", :enabled=\u003etrue}\n```\n\nOr an `Avro::Schema` object can be specified directly:\n\n```ruby\nclass MyModel\n  include Avromatic::Model.build(schema: schema_object)\nend\n```\n\nA specific subject name can be associated with the schema:\n```ruby\nclass MyModel\n  include Avromatic::Model.build(schema_name: 'my_model',\n                                 schema_subject: 'my_model-value')\nend\n```\n\nModels are generated as immutable value\nobjects by default, but can optionally be defined as mutable:\n\n```ruby\nclass MyModel\n  include Avromatic::Model.build(schema_name: :my_model, mutable: true)\nend\n```\n\nGenerated models include attributes for each field in the Avro schema\nincluding any default values defined in the schema.\n\nA model may be defined with both a key and a value schema:\n\n```ruby\nclass MyTopic\n  include Avromatic::Model.build(value_schema_name: :topic_value,\n                                 key_schema_name: :topic_key)\nend\n```\n\nWhen key and value schemas are both specified, attributes are added to the model\nfor the union of the fields in the two schemas.\n\nBy default, optional fields are not allowed in key schemas since their values may\nbe accidentally omitted leading to problems if data is partitioned based on the\nkey values.\n\nThis behavior can be overridden by specifying the `:allow_optional_key_fields`\noption for the model:\n\n```ruby\nclass MyTopic\n  include Avromatic::Model.build(value_schema_name: :topic_value,\n                                 key_schema_name: :topic_key,\n                                 allow_optional_key_fields: true)\nend\n```\n\nA specific subject name can be associated with both the value and key schemas:\n```ruby\nclass MyTopic\n  include Avromatic::Model.build(value_schema_name: :topic_value,\n                                 value_schema_subject: 'topic_value-value',\n                                 key_schema_name: :topic_key,\n                                 key_schema_subject: 'topic_key-value')\nend\n```\n\nA model can also be generated as an anonymous class that can be assigned to a\nconstant:\n\n```ruby\nMyModel = Avromatic::Model.model(schema_name :my_model)\n```\n\n#### Experimental: Union Support\n\nAvromatic contains experimental support for unions containing more than one\nnon-null member type. This feature is experimental because Avromatic\nmay attempt to coerce between types too aggressively.\n\nFor now, if a union contains [nested models](#nested-models) then it is\nrecommended that you assign model instances.\n\nSome combination of the ordering of member types in the union and relying on\nmodel validation may be required so that the correct member is selected,\nespecially when deserializing from Avro.\n\nIn the future, the type coercion used in the gem will be enhanced to better\nsupport the union use case.\n\n#### Nested Models\n\nNested models are models that are embedded within top-level models generated\nusing Avromatic. Normally these nested models are automatically generated.\n\nBy default, nested models are stored in `Avromatic.nested_models`. This is an\n`Avromatic::ModelRegistry` instance that provides access to previously generated\nnested models by the full name of their Avro schema.\n\n```ruby\nAvromatic.nested_models['com.my_company.test.example']\n#=\u003e \u003cmodel class\u003e\n```\n\nThe `ModelRegistry` can be customized to remove a namespace prefix:\n\n```ruby\nAvromatic.nested_models =\n  Avromatic::ModelRegistry.new(remove_namespace_prefix: 'com.my_company')\n```\n\nThe `:remove_namespace_prefix` value can be a string or a regexp.\n\nBy default, top-level generated models reuse `Avromatic.nested_models`. This\nallows nested models to be shared across different generated models.\nA `:nested_models` option can be specified when generating a model. This allows\nthe reuse of nested models to be scoped:\n\n```ruby\nAvromatic::Model.model(schema_name, :my_model\n                       nested_models: ModelRegistry.new)\n```\n\nOnly models without a key schema can be used as nested models. When a model is\ngenerated with just a value schema then it is automatically registered so that\nit can be used as a nested model.\n\nTo extend a model that will be used as a nested model, you must ensure that it\nis defined, which will register it, prior it being referenced by another model.\n\nUsing the `Avromatic.eager_load_models` option allows models that are extended\nand will be used as nested models to be defined at the end of the `.configure`\nblock. In Rails applications, these models are also re-registered after\n`nested_models` is cleared when code reloads to ensure that classes load in the\ncorrect order:\n\n```ruby\nAvromatic.configure do |config|\n  config.eager_load_models = [\n    # reference any extended models that should be defined first\n    'MyNestedModel'\n  ]\nend\n```\n\n#### Custom Type Configuration\n\nCustom types can be configured for fields of named types (record, enum, fixed).\nThese customizations are registered on the `Avromatic` module. Once a custom type\nis registered, it is used for all models with a schema that references that type.\nIt is recommended to register types within a block passed to `Avromatic.configure`:\n\n```ruby\nAvromatic.configure do |config|\n  config.register_type('com.example.my_string', MyString)\nend\n```\n\nThe full name of the type and an optional class may be specified. When a class is\nprovided then values for attributes of that type are defined using the specified\nclass.\n\nIf the provided class responds to the class methods `from_avro` and `to_avro`\nthen those methods are used to convert values when assigning to the model and\nbefore encoding using Avro respectively.\n\n`from_avro` and `to_avro` methods may be also be specified as Procs when\nregistering the type:\n\n```ruby\nAvromatic.configure do |config|\n  config.register_type('com.example.updown_string') do |type|\n    type.from_avro = -\u003e(value) { value.upcase }\n    type.to_avro = -\u003e(value) { value.downcase }\n  end\nend\n```\n\nNil handling is not required as the conversion methods are not be called if the\ninbound or outbound value is nil.\n\nIf a custom type is registered for a record-type field, then any `to_avro`\nmethod/Proc should return a Hash with string keys for encoding using Avro.\n\n### Encoding and Decoding\n\n`Avromatic` provides two different interfaces for encoding the key (optional)\nand value associated with a model.\n\n#### Manually Managed Schemas\n\nThe attributes for the value schema used to define a model can be encoded using:\n\n```ruby\nencoded_value = model.avro_raw_value\n```\n\nIn order to decode this data, a copy of the value schema is required.\n\nIf a model also has an Avro schema for a key, then the key attributes can be\nencoded using:\n\n```ruby\nencoded_key = model.avro_raw_key\n```\n\nIf attributes were encoded using the same schema(s) used to define a model, then\nthe data can be decoded to create a new model instance:\n\n```ruby\nMyModel.avro_raw_decode(key: encoded_key, value: encoded_value)\n```\n\nIf the attributes where encoded using a different version of the model's schemas,\nthen a new model instance can be created by also providing the schemas used to\nencode the data:\n\n```ruby\nMyModel.avro_raw_decode(key: encoded_key,\n                        key_schema: writers_key_schema,\n                        value: encoded_value,\n                        value_schema: writers_value_schema)\n```\n\n#### Messaging API\n\nThe other interface for encoding and decoding attributes uses the\n`AvroTurf::Messaging` API. This interface leverages a schema registry and\nprefixes the encoded data with an id to identify the schema. In this approach,\na schema registry is used to ensure that the correct schemas are available during\ndecoding.\n\nThe attributes for the value schema can be encoded with a schema id prefix using:\n\n```ruby\nmessage_value = model.avro_message_value\n```\n\nIf a model has an Avro schema for a key, then those attributes can also be encoded\nprefixed with a schema id:\n\n```ruby\nmessage_key = model.avro_message_key\n```\n\nA model instance can be created from a key and value encoded in this manner:\n\n```ruby\nMyTopic.avro_message_decode(message_key, message_value)\n```\n\nOr just a value if only one schema is used:\n\n```ruby\nMyValue.avro_message_decode(message_value)\n```\n\nThe schemas associated with a model can also be added to a schema registry without\nencoding a message:\n\n```ruby\nMyTopic.register_schemas!\n```\n\n#### Avromatic::Model::MessageDecoder\n\nA stream of messages encoded from various models using the messaging approach\ncan be decoded using `Avromatic::Model::MessageDecoder`. The decoder must be\ninitialized with the list of models to decode:\n\n```ruby\ndecoder = Avromatic::Model::MessageDecoder.new(MyModel1, MyModel2)\n\ndecoder.decode(model1_messge_key, model1_message_value)\n# =\u003e instance of MyModel1\ndecoder.decode(model2_message_value)\n# =\u003e instance of MyModel2\n```\n\n### Validations and Coercions\n\nAn exception will be thrown if an attribute value cannot be coerced to the corresponding Avro schema field's type.\nThe following coercions are supported:\n\n| Ruby Type | Avro Type |\n| --------- | --------- |\n| String, Symbol | string |\n| Array | array |\n| Hash | map |\n| Integer | int |\n| Integer | long |\n| Float, Integer | float |\n| Float, Integer | double |\n| String | bytes |\n| Date, Time, DateTime | date |\n| Time, DateTime | timestamp-millis |\n| Time, DateTime | timestamp-micros |\n| Float, Integer, BigDecimal | decimal |\n| TrueClass, FalseClass | boolean |\n| NilClass | null |\n| Hash | record |\n\nValidation of required fields is done automatically when serializing a model to Avro. It can also be done\nexplicitly by calling the `valid?` or `invalid?` methods from the\n[ActiveModel::Validations](https://edgeapi.rubyonrails.org/classes/ActiveModel/Validations.html) interface.\n\n### RSpec Support\n\nThis gem also includes an `\"avromatic/rspec\"` file that can be required to support\nusing Avromatic with a fake schema registry during tests.\n\nRequiring this file configures a RSpec before hook that directs any schema\nregistry requests to a fake, in-memory schema registry (instead of port 21001) and rebuilds the\n`Avromatic::Messaging` object for each example.\n\nNote: Use of `avromatic/rspec` requires installing the `sinatra` gem for the in-memory schema registry to work properly.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at https://github.com/salsify/avromatic.\n\n\n## License\n\nThe gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalsify%2Favromatic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsalsify%2Favromatic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsalsify%2Favromatic/lists"}