{"id":16100498,"url":"https://github.com/kurron/avro-experiment","last_synced_at":"2025-04-05T23:41:58.058Z","repository":{"id":31614420,"uuid":"35179437","full_name":"kurron/avro-experiment","owner":"kurron","description":"Taking Apache Avro out for a test spin","archived":false,"fork":false,"pushed_at":"2017-07-05T15:25:56.000Z","size":180,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-12T05:58:41.647Z","etag":null,"topics":["avro","groovy"],"latest_commit_sha":null,"homepage":null,"language":"Groovy","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kurron.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-05-06T19:47:43.000Z","updated_at":"2017-06-14T00:43:21.000Z","dependencies_parsed_at":"2022-09-06T12:40:11.724Z","dependency_job_id":null,"html_url":"https://github.com/kurron/avro-experiment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurron%2Favro-experiment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurron%2Favro-experiment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurron%2Favro-experiment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kurron%2Favro-experiment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kurron","download_url":"https://codeload.github.com/kurron/avro-experiment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247415928,"owners_count":20935384,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","groovy"],"created_at":"2024-10-09T18:46:56.929Z","updated_at":"2025-04-05T23:41:58.042Z","avatar_url":"https://github.com/kurron.png","language":"Groovy","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Overview\nThis project is a simple experiment examining the role that\n[Apache Avro](https://avro.apache.org/) can play in the context of two applications\nthat communicate via message passing.  We'll will simulate the message passing\nby writing messages to disk and having tests reading those files.\n\nOne of Avro's strengths is that can handle many forward and backward\ncompatibility scenarios.  It can do so because each message is associated\nwith a schema that allows the Avro runtime to make decisions about how to\nconvert a payload into an object that the application understands.\n\nIn our test scenario we will have two applications, one that produces the\nmessages and one that consumes them.  Ideally both applications should be\nusing the same message structure but, in practice, that rarely happens.  The\napplications get updated and released on their own schedules so it is important\nto allow each application to deal with message format changes at their own pace.\n\nLuckily, Avro does not require the producer and consumer to use the same\nschema.  Although it is possible to embed a \"pointer\" to a schema inside\neach message, we will assume that each application has a schema embedded\ninside it and only uses that.  Over time, each application will embed different\nrevisions of the same schema.  Our experiment will cover the following\nscenarios:\n\n| Producer      | Consumer      | Notes                                                                                      |\n| ------------- | ------------- | ------------------------------------------------------------------------------------------ |\n| Version 1.0.0 | Version 1.0.0 |                                                                                            |  \n| Version 1.0.0 | Version 1.1.0 | Adds additional field in a forwards compatible way                                         |\n| Version 1.1.0 | Version 1.1.0 |                                                                                            |\n| Version 1.1.0 | Version 1.2.0 | Splits the name field into two fields in a forwards compatible way                         | \n| Version 1.2.0 | Version 1.2.0 |                                                                                            |\n| Version 1.2.0 | Version 1.3.0 | Adds complex types, such as arrays, maps and promotable types in a forwards compatible way | \n| Version 1.3.0 | Version 1.3.0 |                                                                                            |\n| Version 1.3.0 | Version 1.4.0 | Promotes the types, eg. int to long in a forwards compatible way                           |\n| Version 2.0.0 | Version 2.0.0 |                                                                                            |\n| Version 1.4.0 | Version 2.0.0 | Removes one field and adds another one in a forwards incompatible way                      | \n\nThe schema version uses [Semantic Versioning](http://semver.org/) to indicate\nbreaking and non-breaking changes.\n\n## Definitions\n* **Backward Compatibility** - the writer is using a newer schema than the reader \n* **Forward Compatibility** - the writer is using an older schema than the reader \n* **Backward Compatibility** - the reader is using an older schema than the writer \n* **Forward Compatibility** - the reader is using a newer schema than the writer \n \n# Prerequisites\n\n* [JDK](http://www.oracle.com/technetwork/java/index.html) installed and working\n\n# Building\nUse `./gradlew` to execute the [Gradle](https://gradle.org/) build script.\n\n# Installation\n* [Avro Tools](http://avro.apache.org/releases.html) downloaded into the project directory\n\n# Tips and Tricks\n\n## Jackson's Avro Support\nInitial testing was done using [Jackson's Avro support](https://github.com/FasterXML/jackson-dataformats-binary/tree/master/avro)\nbut it was quickly found that it does not support default values which is required\nto maintain forward compatibility.  For that reason, the test code has been removed and\ntesting continued using the native Avro library.\n\n## Avro Code Generation\nThe tests were written using Avro's optional code generation facilities.  Although\nit is possible to use Avro in a less structured way, via untyped key-vales, it\nis assumed that application developers would prefer to use typed structures.\n\n## Avro Inconveniences\nThe generated Avro structures do not use native JVM strings and, instead, use either\na custom UTF-8 class or `java.lang.CharSequence`.  For this reason, the tests\ncontain conversions that you might find odd.\n\n## How We Test\nEach schema revision must live in its own module because the schema's namespace\nmust remain constant or the compatibility conversions will not be applied.\nFor example, changing the namespace from `org.kurron.avro.example` to\n`org.kurron.avro.example.v100` would, in Avro's mind, create two separate entities\nand it would not attempt a conversion.\n\nWe are counting on Gradle's current behavior of building the modules in alphabetical\norder.  This is required because the input of a test is the output file of the previous\nmodule.  For example, the v130 test attempts to read the v120 file when testing forwards compatibility.\n\n## Interesting Avro Features\n1. You can rename a field via the `aliases` construct.\n1. Providing a default value for a field, via the `default` construct, guarantees forward compatibility.\n1. Fields of type `int`, `long`, `float`, `string` and `byte` can be changed in a compatible way.\n1. Rich constructs, such as `records`, `array`, `maps`, `enums` and `unions` provide many possible structures.\n1. Logical types, including `Date`, `Time` and `Duration` exist.\n1. Batch processing is supported via files.\n1. RPC messaging is also supported.\n\n## Backwards Compatibility Testing\nTo be complete, we tested backward compatibility scenarios.  For this experiment, we had to switch away\nfrom generated, type-safe object and used generic key-value maps instead.\n\n| Producer      | Consumer      | Notes                                                 |\n| ------------- | ------------- | ------------------------------------------------------|\n| Version 1.1.0 | Version 1.0.0 |                                                       |  \n| Version 1.2.0 | Version 1.0.0 |                                                       |\n| Version 1.2.0 | Version 1.1.0 |                                                       |\n| Version 1.3.0 | Version 1.0.0 |                                                       |\n| Version 1.3.0 | Version 1.1.0 |                                                       |\n| Version 1.3.0 | Version 1.2.0 |                                                       |\n| Version 1.4.0 | Version 1.0.0 |                                                       |\n| Version 1.4.0 | Version 1.1.0 |                                                       |\n| Version 1.4.0 | Version 1.2.0 |                                                       |\n| Version 1.4.0 | Version 1.3.0 | The promotion from int to a long breaks compatibility |\n\n## Self-Describing Data\nAvro's ability to apply schema compatibility rules via the generated code is a real\ntime saver. It isn't perfect, however.  As our testing confirmed, there are cases where\nthe schema change is too great and Avro is unable to read in the data.  Although complex,\nit is possible to read in the data in a non-type-safe way and \"pick\" out the desired\nattributes by hand, applying migration rules in your own code.  One way to do this\nis by embedding a reference to the schema with the data. \n\n```json\n{\n   \"schema\":\"s3://kurron-schemas/foo/v100\",\n   \"data\":{\n      \"factory\":\"Factory A\",\n      \"serialNumber\":\"EU3571\",\n      \"status\":\"RUNNING\",\n      \"lastStartedAt\":1474141826926,\n      \"temperature\":34.56,\n      \"endOfLife\":false,\n      \"floorNumber\":{\n         \"int\":2\n      }\n   }\n}\n```\n\nThe application would consume the JSON, dereference the `schema` attribute and read\nthe `data` attribute with that schema.  In a RabbitMQ setting, the AMQP protocol has\nthe `type` header which can be used to hold the schema reference to the binary payload.\n\nThe benefit of perfect deserialization coupled with by-hand migration rules must be\nquestioned. The application must be updated each time an unknown schema is encountered.\nThis is not the case when using Avro generated data objects.  Perhaps automated testing\nof any newly generated schema is a better solution?  We've essentially done that in this\nproject and ideas could be refined into something that could live in a CI/CD pipeline.\nAt least the author of the change would know that she is creating a breaking change.\n\n## Serialization Notes\nThese tests used the `DataFileWriter` to encode data to disk which worked fine in\nthis context but how do we serialize to an in-memory representation?  We need to\ndo that if Avro is being used in RabbitMQ or REST payloads.  It took me a while\nbut I found a technique.\n\n```groovy\ndef schema = new Schema.Parser().parse(DatFileWriter.getResourceAsStream('/schema/user.json'))\ndef factory = EncoderFactory.get()\ndef stream = new ByteArrayOutputStream()\ndef encoder = factory.jsonEncoder( schema, stream, true )\ndef writer = new SpecificDatumWriter\u003cUser\u003e( User )\nwriter.write( user, encoder )\nencoder.flush()\nprintln stream\n```\n\nThe above sample encodes the type-safe object into an Avro JSON format.  The\nbinary format can by used simply by swapping out the encoder.\n\n```groovy\ndef binaryEncoder = factory.directBinaryEncoder( stream, null )\ndef encoder = factory.validatingEncoder( schema, binaryEncoder )\n```\n\nTo read an in-memory stream we can do something similar to this:\n\n```groovy\ndef decoderFactory = DecoderFactory.get()\ndef inputStream = new ByteArrayInputStream( buffer )\ndef binaryDecoder = decoderFactory.directBinaryDecoder( inputStream, null )\ndef decoder = decoderFactory.validatingDecoder( schema, binaryDecoder )\ndef reader = new SpecificDatumReader\u003cUser\u003e( schema, schema )\ndef user = new User()\nreader.read( user, decoder )\n```\n\nTo read from a JSON encoded stream, swap out the decoder:\n\n```groovy\ndef jsonDecoder = decoderFactory.jsonDecoder( schema, inputStream )\ndef decoder = decoderFactory.validatingDecoder( schema, jsonDecoder )\n```\n\nMy experiments show that the application not only has to know that schema that\nwas used to write the data but **also the encoding that was used**.  Reading\nbinary encoded data using a JSON decoder does not work.  This means that\na self-describing message must also specify the encoding format as\nwell as the writer's schema.\n\n# Troubleshooting\n\n# License and Credits\nThis project is licensed under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/).\n\n* [Event Streams in Action: Unified log processing with Kafka and Kinesis](https://www.manning.com/books/event-streams-in-action)\n* [Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems](http://shop.oreilly.com/product/0636920032175.do)\n* [Kafka Streams in Action](https://www.manning.com/books/kafka-streams-in-action)\n* [Streaming Data: Understanding the real-time pipeline](https://www.manning.com/books/streaming-data)\n* [Big Data: Principles and best practices of scalable realtime data systems](https://www.manning.com/books/big-data)\n* [Kafka The Definitive Guide: Real-Time Data and Stream Processing at Scale](http://shop.oreilly.com/product/0636920044123.do)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkurron%2Favro-experiment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkurron%2Favro-experiment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkurron%2Favro-experiment/lists"}