{"id":19468799,"url":"https://github.com/imrafaelmerino/avro-spec","last_synced_at":"2026-02-16T14:36:05.016Z","repository":{"id":263006853,"uuid":"848704811","full_name":"imrafaelmerino/avro-spec","owner":"imrafaelmerino","description":"avro-spec empowers you to create Avro schemas  with the specs from json-values. Leveraging the simplicity and composability of creating specs allows you to efficiently define Avro schemas. The provided serializers/deserializers enable the transmission of the immutable and persistent JSON from json-values through the wire in Avro format","archived":false,"fork":false,"pushed_at":"2024-11-23T21:50:24.000Z","size":686,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-06T22:29:36.742Z","etag":null,"topics":["avro","avro-schema","confluent","confluent-schema-registry","java","json","json-values","kafka","kafka-deserializer","kafka-serializer","persistent-data-structure"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/imrafaelmerino.png","metadata":{"files":{"readme":"readme.md","changelog":"changelog.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-28T08:53:23.000Z","updated_at":"2024-11-23T21:50:27.000Z","dependencies_parsed_at":"2024-11-15T15:29:20.284Z","dependency_job_id":"b33f0ae9-3c6e-4858-a5e2-2bda0a702da4","html_url":"https://github.com/imrafaelmerino/avro-spec","commit_stats":null,"previous_names":["imrafaelmerino/avro-spec"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/imrafaelmerino/avro-spec","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imrafaelmerino%2Favro-spec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imrafaelmerino%2Favro-spec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imrafaelmerino%2Favro-spec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imrafaelmerino%2Favro-spec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/imrafaelmerino","download_url":"https://codeload.github.com/imrafaelmerino/avro-spec/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/imrafaelmerino%2Favro-spec/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29510292,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-16T09:05:14.864Z","status":"ssl_error","status_checked_at":"2026-02-16T08:55:59.364Z","response_time":115,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avro","avro-schema","confluent","confluent-schema-registry","java","json","json-values","kafka","kafka-deserializer","kafka-serializer","persistent-data-structure"],"created_at":"2024-11-10T18:43:56.704Z","updated_at":"2026-02-16T14:36:04.997Z","avatar_url":"https://github.com/imrafaelmerino.png","language":"Java","funding_links":["https://www.buymeacoffee.com/imrafaelmerino"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./avro-spec-logo.png\" alt=\"Logo\" width=\"300\" height=\"300\" /\u003e\n\u003c/p\u003e\n\n[![Maven](https://img.shields.io/maven-central/v/com.github.imrafaelmerino/avro-spec/1.0.0)](https://search.maven.org/artifact/com.github.imrafaelmerino/avro-spec/1.0.0/jar)\n\n[![Buy Me a Coffee](https://img.shields.io/badge/Buy%20Me%20a%20Coffee-%E2%98%95%20Support-yellow)](https://www.buymeacoffee.com/imrafaelmerino)\n\n- [Avro spec](#avro-spec)\n- [Avro schemas](#avro-schema)\n- [A more elaborated example with recursive schemas](#recursive-schema)\n- [Avro serializer and deserializer](#seria-deseria)\n- [Installation](#installation)\n- [Running Integration Tests](#rit)\n\n## \u003ca name=\"avro-spec\"\u003e\u003ca/\u003e Avro spec\n\n`avro-spec` empowers you to create [Avro](https://avro.apache.org/) schemas and\nserializers/deserializers with the [specs](https://github.com/imrafaelmerino/json-values#specs) from\n[json-values](https://github.com/imrafaelmerino/json-values). Leveraging the simplicity,\nintuitiveness, and composability of creating specs allows you to efficiently define Avro schemas.\nThe provided serializers/deserializers enable the transmission of the immutable and persistent JSON\nfrom [json-values](https://github.com/imrafaelmerino/json-values) through the wire in Avro format,\nsupporting Confluent Schema Registry.\n\n## \u003ca name=\"avro-schema\"\u003e\u003ca/\u003e Avro schemas\n\nCreate Avro schemas in a natural way, for example, from the `personSpec`:\n\n```java\n\nJsSpec typeEmailSpec = JsEnumBuilder.withName(\"phone_type\")\n                                    .withNamespace(\"example.com\")\n                                    .build(\"MOBILE\",\"FIXED\");\n\nJsSpec phoneSpec = JsObjSpecBuilder.withName(\"person_phone\")\n                                   .withNamespace(\"example.com\")\n                                   .build(JsObjSpec.of(\"number\", JsSpecs.str(),\n                                                       \"type\", typeEmailSpec\n                                                      )\n                                         );\n\nJsSpec emailSpec = JsObjSpecBuilder.withName(\"person_email\")\n                                   .withNamespace(\"example.com\")\n                                   .build(JsObjSpec.of(\"address\", JsSpecs.str(),\n                                                       \"verified\", JsSpecs.bool()\n                                                      )\n                                         );\n\nJsSpec contactSpec = JsSpecs.oneSpecOf(phoneSpec, emailSpec);\n\n\nJsObjSpec personSpec =\n    JsObjSpecBuilder.withName(\"person\")\n                    .withNamespace(\"example.com\")\n                    .build(JsObjSpec.of(\"name\", JsSpecs.str(),\n                                        \"age\", JsSpecs.integer(n -\u003e n \u003e 16),\n                                        \"surname\", JsSpecs.str(),\n                                        \"contact\", contactSpec\n                                       )\n                          );\n\nSchema schema = SpecToSchema.convert(personSpec);\n\nSystem.out.println(schema);\n\n```\n\nResulting Schema:\n\n```json\n{\n  \"type\": \"record\",\n  \"name\": \"person\",\n  \"namespace\": \"example.com\",\n  \"fields\": [\n    {\n      \"name\": \"name\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"age\",\n      \"type\": \"int\"\n    },\n    {\n      \"name\": \"surname\",\n      \"type\": \"string\"\n    },\n    {\n      \"name\": \"contact\",\n      \"type\": [\n        {\n          \"type\": \"record\",\n          \"name\": \"person_phone\",\n          \"fields\": [\n            {\n              \"name\": \"number\",\n              \"type\": \"string\"\n            },\n            {\n              \"name\": \"type\",\n              \"type\": {\n                \"type\": \"enum\",\n                \"name\": \"phone_type\",\n                \"symbols\": [\"MOBILE\", \"FIXED\"]\n              }\n            }\n          ]\n        },\n        {\n          \"type\": \"record\",\n          \"name\": \"person_email\",\n          \"fields\": [\n            {\n              \"name\": \"address\",\n              \"type\": \"string\"\n            },\n            {\n              \"name\": \"verified\",\n              \"type\": \"boolean\"\n            }\n          ]\n        }\n      ]\n    }\n  ]\n}\n```\n\n## \u003ca name=\"recursive-schema\"\u003e\u003ca/\u003e More elaborated example with recursive schemas\n\nThe [json-values](https://github.com/imrafaelmerino/json-values/) library simplifies the\nimplementation of inheritance and the generation of structured data in JSON. Let's explore an\nexample showcasing the ease of defining object specifications and its Avro schema, generating data,\nand validating against specifications.\n\nIn this example, picked from [this\narticle](https://json-schema.org/blog/posts/modelling-inheritance#so-is-inheritance-in-json-schema-possible)\nwe model a hierarchy of devices, including mice, keyboards, and USB hubs. Each device type has\nspecific attributes, and we use inheritance to share common fields across all device types.\n\n```java\n\n    String NAME_FIELD = \"name\";\n    String TYPE_FIELD = \"type\";\n    String BUTTON_COUNT_FIELD = \"buttonCount\";\n    String WHEEL_COUNT_FIELD = \"wheelCount\";\n    String TRACKING_TYPE_FIELD = \"trackingType\";\n    String KEY_COUNT_FIELD = \"keyCount\";\n    String MEDIA_BUTTONS_FIELD = \"mediaButtons\";\n    String CONNECTED_DEVICES_FIELD = \"connectedDevices\";\n    String PERIPHERAL_FIELD = \"peripheral\";\n    List\u003cString\u003e TRACKING_TYPE_ENUM = List.of(\"ball\",\n                                              \"optical\");\n\n    JsObjSpec baseSpec =\n        JsObjSpec.of(NAME_FIELD,\n                     JsSpecs.str()\n                    );\n\n    JsObjSpec mouseSpec =\n        JsObjSpecBuilder.withName(\"mouse\")\n                        .build(JsObjSpec.of(BUTTON_COUNT_FIELD,\n                                            JsSpecs.integer(),\n                                            TYPE_FIELD,\n                                            JsSpecs.cons(\"mouse_type\",\n                                                         JsStr.of(\"mouse\")),\n                                            WHEEL_COUNT_FIELD,\n                                            JsSpecs.integer(),\n                                            TRACKING_TYPE_FIELD,\n                                            JsEnumBuilder.withName(\"tracking_type\")\n                                                         .build(TRACKING_TYPE_ENUM)\n                                           )\n                                        .concat(baseSpec)\n                              );\n\n    JsObjSpec keyboardSpec =\n        JsObjSpecBuilder.withName(\"keyboard\")\n                        .build(JsObjSpec.of(KEY_COUNT_FIELD,\n                                            JsSpecs.integer(),\n                                            TYPE_FIELD,\n                                            JsSpecs.cons(\"keyboard_type\",\n                                                         JsStr.of(\"keyboard\")),\n                                            MEDIA_BUTTONS_FIELD,\n                                            JsSpecs.bool()\n                                           )\n                                        .concat(baseSpec)\n                              );\n\n    JsObjSpec usbHubSpec =\n        JsObjSpecBuilder.withName(\"usb_hub\")\n                        .withFieldsDefaults(Map.of(CONNECTED_DEVICES_FIELD,\n                                                   JsNull.NULL))\n                        .build(JsObjSpec.of(TYPE_FIELD,\n                                            JsSpecs.cons(\"usb_hub_type\",\n                                                         JsStr.of(\"usb_hub\")),\n                                            CONNECTED_DEVICES_FIELD,\n                                            JsSpecs.arrayOfSpec(JsSpecs.ofNamedSpec(PERIPHERAL_FIELD))\n                                                   .nullable()\n                                           )\n                                        .withOptKeys(CONNECTED_DEVICES_FIELD)\n                                        .concat(baseSpec)\n                              );\n\n    JsSpec peripheralSpec =\n        JsSpecs.ofNamedSpec(PERIPHERAL_FIELD,\n                            JsSpecs.oneSpecOf(mouseSpec,\n                                              keyboardSpec,\n                                              usbHubSpec\n                                             )\n                           );\n\n    Schema schema = SpecToAvroSchema.convert(peripheralSpec);\n\n    System.out.println(schema);\n\n```\n\nand the Avro schema would be:\n\n```json\n[\n  {\n    \"type\": \"record\",\n    \"name\": \"mouse\",\n    \"fields\": [\n      {\n        \"name\": \"wheelCount\",\n        \"type\": \"int\"\n      },\n      {\n        \"name\": \"name\",\n        \"type\": \"string\"\n      },\n      {\n        \"name\": \"buttonCount\",\n        \"type\": \"int\"\n      },\n      {\n        \"name\": \"type\",\n        \"type\": {\n          \"type\": \"enum\",\n          \"name\": \"mouse_type\",\n          \"symbols\": [\"mouse\"]\n        }\n      },\n      {\n        \"name\": \"trackingType\",\n        \"type\": {\n          \"type\": \"enum\",\n          \"name\": \"tracking_type\",\n          \"symbols\": [\"ball\", \"optical\"]\n        }\n      }\n    ]\n  },\n  {\n    \"type\": \"record\",\n    \"name\": \"keyboard\",\n    \"fields\": [\n      {\n        \"name\": \"name\",\n        \"type\": \"string\"\n      },\n      {\n        \"name\": \"keyCount\",\n        \"type\": \"int\"\n      },\n      {\n        \"name\": \"mediaButtons\",\n        \"type\": \"boolean\"\n      },\n      {\n        \"name\": \"type\",\n        \"type\": {\n          \"type\": \"enum\",\n          \"name\": \"keyboard_type\",\n          \"symbols\": [\"keyboard\"]\n        }\n      }\n    ]\n  },\n  {\n    \"type\": \"record\",\n    \"name\": \"usb_hub\",\n    \"fields\": [\n      {\n        \"name\": \"type\",\n        \"type\": {\n          \"type\": \"enum\",\n          \"name\": \"usb_hub_type\",\n          \"symbols\": [\"usb_hub\"]\n        }\n      },\n      {\n        \"name\": \"name\",\n        \"type\": \"string\"\n      },\n      {\n        \"name\": \"connectedDevices\",\n        \"type\": [\n          \"null\",\n          {\n            \"type\": \"array\",\n            \"items\": [\"mouse\", \"keyboard\", \"usb_hub\"]\n          }\n        ],\n        \"default\": null\n      }\n    ]\n  }\n]\n```\n\n## \u003ca name=\"conversion\"\u003e\u003ca/\u003e Convert Json to Avro objects and vice versa\n\nTable 1: Avro Type Mappings\n\n| Avro Type | json-values | Avro class                                     |\n| --------- | ----------- | ---------------------------------------------- |\n| null      | JsNull.Null | null                                           |\n| boolean   | JsBool      | java.lang.Boolean                              |\n| int       | JsInt       | java.lang.Integer                              |\n| long      | JsLong      | java.lang.Long                                 |\n| float     | JsDouble    | java.lang.Float                                |\n| double    | JsDouble    | java.lang.Double                               |\n| bytes     | JsBinary    | java.nio.HeapByteBuffer                        |\n| string    | JsStr       | java.lang.String                               |\n| record    | JsObj       | org.apache.avro.generic.GenericData$Record     |\n| enum      | JsStr       | org.apache.avro.generic.GenericData$EnumSymbol |\n| array     | JsArray     | org.apache.avro.generic.GenericData$Array      |\n| map       | JsObj       | java.util.HashMap                              |\n| fixed     | JsBinary    | org.apache.avro.generic.GenericData$Fixed      |\n\navro-spec defines the following logical types to serialize and deserialize big integers represented\nwith `JsBigDec`, big decimal represented with `JsBigInt`, and instants represented with `JsInstant`:\n\n| Avro Type | Logical Type | json-values Type | Avro class       |\n| --------- | ------------ | ---------------- | ---------------- |\n| string    | bigdecimal   | JsBigDec         | java.lang.String |\n| string    | biginteger   | JsBigInt         | java.lang.String |\n| string    | iso-8601     | JsInstant        | java.lang.String |\n\nLet's see how to do conversions between json-values and Avro objects.\n\nFrom JSON to Avro using specs\n\n```java\n\nJsObj obj = ???;\nJsSpec objSpec = ???;\nGenericData.Record record = JsonToAvro.convert(obj, objSpec);\n\nJsArray array = ???;\nJsSpec arrSpec = ???;\nGenericData.Array\u003cObject\u003e = JsonToAvro.convert(array, arrSpec);\n\nJson\u003c?\u003e json = ???;\nJsSpec jsonSpec = ???;\nGenericContainer container = JsonToAvro.convert(json, jsonSpec);\n\n```\n\nFrom JSON to Avro using Avro schemas\n\n```java\n\nJsObj obj = ???;\nSchema objSchema = ???;\nGenericData.Record record = JsonToAvro.convert(obj, objSchema);\n\nJsArray array = ???;\nSchema arrSchema = ???;\nGenericData.Array\u003cObject\u003e = JsonToAvro.convert(array, arrSchema);\n\nJson\u003c?\u003e json = ???;\nJsSpec jsonSpec = ???;\nGenericContainer container = JsonToAvro.convert(json, jsonSpec);\n\nJsValue value = ???;\nSchema valueSchema = ???;\nObject object = JsonToAvro.convertValue(value, valueSchema);\n\n```\n\nAnd from JSON to Avro:\n\n```java\nGenericData.Record record = ???;\nJsObj obj = AvroToJson.convert(record);\n\nGenericData.Array\u003cObject\u003e avroArr = ???;\nJsArray arr = AvroToJson.convert(avroArr);\n\nObject avroObj = ???;\nSchema avroObjSchema = ???;\nJsValue value = AvroToJson.convert(avroObj, avroObjSchema);\n\nMap\u003c?, ?\u003e map = ???;\nSchema mapSchema = ???;\nJsObj mapObj = AvroToJson.convert(map, mapSchema);\n\n```\n\n## \u003ca name=\"serializers\"\u003e\u003ca/\u003e Avro serializers and deserializers\n\nAvro serializers and deserializers play a crucial role in efficiently encoding and decoding data for\ncommunication between Kafka producers and consumers. Here, we explore different options available\nfor serialization and deserialization with Avro, including integration with the Confluent Schema\nRegistry.\n\n- Confluent Avro Serializers with Schema Registry Integration\n\n  - `jsonvalues.spec.serializers.confluent.ConfluentSerializer`: Serializes Avro generic containers\n    of type `org.apache.avro.generic.GenericContainer` into bytes.\n  - `jsonvalues.spec.serializers.confluent.ConfluentSpecSerializer`: Serializes a JSON object\n    conforming to a spec into bytes.\n\n- Avro serializers:\n  - `jsonvalues.spec.serializers.SpecSerializer`: Serializes a JSON object conforming to a spec into\n    bytes.\n\nAnd the following deserializers:\n\n- Confluent Avro Deserializers with Schema Registry Integration\n\n  - `jsonvalues.spec.deserializers.confluent.ConfluentObjDeserializer`: Deserializes bytes into a\n    JsObj.\n  - `jsonvalues.spec.deserializers.confluent.ConfluentArrayDeserializer`: Deserializes bytes into a\n    JsArray.\n  - `jsonvalues.spec.deserializers.confluent.ConfluentDeserializer`: Deserializes bytes into a JSON\n    object.\n  - `jsonvalues.spec.deserializers.confluent.ConfluentObjSpecDeserializer`: Deserializes bytes into\n    a JsObj conforming to a spec.\n  - `jsonvalues.spec.deserializers.confluent.ConfluentArraySpecDeserializer`: Deserializes bytes\n    into a JsArray conforming to a spec.\n\n- Avro deserializers:\n  - `jsonvalues.spec.deserializers.ObjSpecDeserializer`: Deserializes bytes into a JsObj conforming\n    to a spec.\n  - `jsonvalues.spec.deserializers.ArraySpecDeserializer`: Deserializes bytes into a JsArray\n    conforming to a spec.\n\n**Which serializer to use?**\n\nLet's consider the following example where we have a topic named \"payments\":\n\n```java\n String TOPIC = \"payments\";\n\n JsObjSpec paymentSpec =\n    JsObjSpecBuilder.withName(\"Payment\")\n                    .withNamespace(\"examples.avro.spec\")\n                    .build(JsObjSpec.of(\"id\", JsSpecs.str(),\n                                        \"amount\", JsSpecs.doubleNumber()\n                                       )\n                          );\n\n //Let's create a generator for payments\n Supplier\u003cJsObj\u003e paymentGen =\n    JsObjGen.of(\"id\", JsStrGen.alphabetic(),\n                \"amount\", JsDoubleGen.arbitrary(100.0d,1000d)\n               )\n            .sample();\n\n```\n\nWhen working with Kafka, **using a single producer for all topics** is highly recommended for\noptimal performance due to its thread safety and batching capabilities. Given that, you have two\noptions:\n\n1. `jsonvalues.spec.confluent.ConfluentSerializer` for Avro Format with Schema Registry Integration:\n\n```java\n private static KafkaProducer\u003cString, GenericRecord\u003e createProducer() {\n    Properties props = new Properties();\n    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\n              StringSerializer.class);\n    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\n              ConfluentSerializer.class);\n    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaProducer\u003c\u003e(props);\n  }\n\n\n int RECORDS = 10;\n try (var producer = createProducer()) {\n    for (long i = 0; i \u003c RECORDS; i++)\n    {\n       JsObj payment = paymentGen.get();\n\n       GenericRecord record =  JsonToAvro.convert(payment,paymentSpec)\n\n       ProducerRecord\u003cString, GenericRecord\u003e record =\n            new ProducerRecord\u003c\u003e(TOPIC,\n                                 payment.getStr(\"id\") + i,\n                                 record);\n\n       producer.send(record);\n       Thread.sleep(1000L);\n   }\n\n   producer.flush();\n   System.out.printf(\"Successfully produced 10 messages to a topic called %s%n\",\n                     TOPIC\n                    );\n\n }\n\n```\n\nIn the first case, we're using `jsonvalues.spec.confluent.ConfluentSerializer` as the value\nserializer for the producer. Before sending the data to the Kafka topic, we need to convert the JSON\nobject (`payment`) into the Avro object `GenericRecord` using\n`JsonToAvro.convert(payment, paymentSpec)`.\n\n2. `jsonvalues.spec.serializers.SpecSerializer` for Avro Format without Schema Registry integration:\n\n```java\n private static KafkaProducer\u003cString, byte[]\u003e createProducer() {\n    Properties props = new Properties();\n    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\n              StringSerializer.class);\n    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\n              ByteArraySerializer.class);\n    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaProducer\u003c\u003e(props);\n }\n\n JsSpecSerializer paymentSerializer =\n        JsSpecSerializerBuilder.of(paymentSpec)\n                               .build();\n\n int RECORDS = 10;\n try (var producer = createProducer()) {\n   for (long i = 0; i \u003c RECORDS; i++)\n   {\n       JsObj payment = paymentGen.get();\n\n       ProducerRecord\u003cString, byte[]\u003e record =\n            new ProducerRecord\u003c\u003e(TOPIC,\n                                 payment.getStr(\"id\") + i,\n                                 paymentSerializer.serialize(payment));\n       producer.send(record);\n       Thread.sleep(1000L);\n   }\n\n   producer.flush();\n   System.out.printf(\"Successfully produced 10 messages to a topic called %s%n\",\n                     TOPIC);\n\n  }\n\n```\n\nIn this second case, we're utilizing `org.apache.kafka.common.serialization.ByteArraySerializer`\nfrom Kafka as the value serializer for the producer. Additionally, we're employing the created\npaymentSerializer to convert the JSON object into bytes in avro format before sending it to the\nKafka topic.\n\nIf you have **one specific producer for a topic**, because the topic requires a specific\nconfiguration:\n\n1. `jsonvalues.spec.serializers.confluent.ConfluentSpecSerializer` for Avro format and integration\n   with Confluent Schema Registry In this case you need to create a new class and extend\n   `JsSpecSerializer` providing the spec. Find below and example:\n\n```java\n\npublic final class PaymentSerializer extends ConfluentSpecSerializer {\n\n  @Override\n  protected boolean isJFREnabled() {\n    return true;\n  }\n\n  @Override\n  protected JsSpec getSpec() {\n    return paymentSpec;\n  }\n}\n\nprivate static KafkaProducer\u003cString, JsObj\u003e createPaymentProducer() {\n\n    Properties props = new Properties();\n    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\n              StringSerializer.class);\n    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\n              PaymentSerializer.class);\n    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaProducer\u003c\u003e(props);\n  }\n\n try (var producer = createPaymentProducer()) {\n    for (long i = 0; i \u003c RECORDS; i++)\n    {\n        JsObj payment = gen.get();\n        ProducerRecord\u003cString, JsObj\u003e record =\n            new ProducerRecord\u003c\u003e(TOPIC,\n                                 payment.getStr(\"id\") + i,\n                                 payment);\n        producer.send(record);\n        Thread.sleep(1000L);\n      }\n\n      producer.flush();\n      System.out.printf(\"Successfully produced 10 messages to a topic called %s%n\",\n                        TOPIC);\n\n    }\n```\n\nIn this setup, you create a new class `PaymentSerializer` by extending `ConfluentSpecSerializer`,\nwhere you override the `getSpec()` method to provide the necessary spec (`paymentSpec`). Then, when\ncreating the Kafka producer (`createPaymentProducer()`), you specify `PaymentSerializer.class` as\nthe value serializer to use for serialization. This serializer will automatically handle the\nserialization process when you pass a `JsObj` to the producer, simplifying the process for you.\n\n2. `jsonvalues.spec.serializers.SpecSerializer` for Avro format without Schema registry. In this\n   case, we use the `org.apache.kafka.common.serialization.ByteArraySerializer` from Kafka as the\n   value serializer for the producer. Additionally, we employ the `JsSpecSerializer` created using a\n   builder to convert JSON objects into bytes in Avro format before sending them to the Kafka topic.\n\n```java\nprivate static KafkaProducer\u003cString, byte[]\u003e createProducer() {\n    Properties props = new Properties();\n    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,\n              StringSerializer.class);\n    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,\n              ByteArraySerializer.class);\n    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaProducer\u003c\u003e(props);\n  }\n\nSpecSerializer paymentSerializer =\n        SpecSerializerBuilder.of(Specs.paymentSpec)\n                             .build();\n\n\nint RECORDS = 10;\ntry (var producer = createProducer())\n    {\n      for (long i = 0; i \u003c RECORDS; i++) {\n        JsObj payment = paymentGen.get();\n\n        ProducerRecord\u003cString, byte[]\u003e record =\n            new ProducerRecord\u003c\u003e(TOPIC,\n                                 payment.getStr(\"id\") + i,\n                                 paymentSerializer.serialize(payment));\n        producer.send(record);\n        Thread.sleep(1000L);\n      }\n\n      producer.flush();\n      System.out.printf(\"Successfully produced 10 messages to a topic called %s%n\",\n                        TOPIC);\n\n    }\n\n\n```\n\nWhat about deserializers?\n\n1. If you opt for the Confluent serializer, you have several deserialization options integrated with\n   the Confluent Schema Registry:\n\n- `jsonvalues.spec.confluent.ConfluentObjDeserializer`: Deserializes into a JsObj.\n- `jsonvalues.spec.confluent.ConfluentArrayDeserializer`: Deserializes into a JsArray.\n- `jsonvalues.spec.confluent.ConfluentDeserializer`: Deserializes into a Json (can be either a JsObj\n  or a JsArray).\n- You can also create custom deserializers by extending\n  `jsonvalues.spec.confluent.ConfluentObjSpecDeserializer` or\n  `jsonvalues.spec.confluent.ConfluentArraySpecDeserializer` to ensure deserialized data conforms to\n  a specific schema.\n\nExample using `ConfluentObjDeserializer`:\n\n```java\n\nprivate static KafkaConsumer\u003cString, JsObj\u003e createPaymentDeserializer() {\n    Properties props = new Properties();\n    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(ConsumerConfig.GROUP_ID_CONFIG,\n              \"my-consumer-group\");\n    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,\n              StringDeserializer.class);\n    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,\n              jsonvalues.spec.confluent.ConfluentObjDeserializer.class);\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaConsumer\u003c\u003e(props);\n  }\n\n\ntry (var consumer = createConsumerWithJsonObjDeserializer()) {\n      consumer.subscribe(List.of(TOPIC));\n      while (true) {\n        ConsumerRecords\u003cString, JsObj\u003e records = consumer.poll(Duration.ofMillis(500));\n        System.out.println(\"Consumed \" + records.count() + \" records.\");\n        for (var record : records) {\n          JsObj obj = record.value();\n          System.out.printf(\"offset = %d, key = %s, value = %s%n\",\n                            record.offset(),\n                            record.key(),\n                            obj\n                           );\n        }\n      }\n    }\n\n```\n\nExample using a spec deserializer:\n\n```java\n\n//must create a new deserializer extending ConfluentObjSpecDeserializer\npublic final class PaymentDeserializer extends ConfluentObjSpecDeserializer {\n\n  @Override\n  protected JsSpec getSpec() {\n    return paymentSpec;\n  }\n\n  @Override\n  protected boolean isJFREnabled() {\n    return true;\n  }\n}\n\nprivate static KafkaConsumer\u003cString, JsObj\u003e createPaymentDeserializer() {\n    Properties props = new Properties();\n    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(ConsumerConfig.GROUP_ID_CONFIG,\n              \"my-consumer-group\");\n    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,\n              StringDeserializer.class);\n    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,\n              PaymentDeserializer.class);\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaConsumer\u003c\u003e(props);\n}\n\n\n// same code as before\n\n\n```\n\nIn this second example, it's important to note that every deserialized object adheres to the payment\nspecification; otherwise, the deserialization process would not succeed.\n\n2. If you're using the serializer without Schema Registry integration, you can employ the builders\n   `jsonvalues.spec.deserializers.ObjSpecDeserializerBuilder` and\n   `jsonvalues.spec.deserializers.ArraySpecDeserializerBuilder` to construct deserializers from\n   specifications. These builders facilitate the creation of custom deserializers tailored to your\n   specific data schemas.\n\nBelow is an example demonstrating the usage of a `ByteArrayDeserializer` from Kafka along with a\nspec deserializer to convert bytes into a JsObj and consume a Kafka topic:\n\n```java\n\n  private static KafkaConsumer\u003cString, Bytes\u003e createConsumer() {\n    Properties props = new Properties();\n    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,\n              \"localhost:29092\");\n    props.put(ConsumerConfig.GROUP_ID_CONFIG,\n              \"group-json-deserializer-1\");\n    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,\n              StringDeserializer.class);\n    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,\n              BytesDeserializer.class);\n    props.put(SCHEMA_REGISTRY_URL_CONFIG,\n              \"http://localhost:8081\");\n    return new KafkaConsumer\u003c\u003e(props);\n  }\n\n  ObjSpecDeserializer paymentDeserializer =\n        ObjSpecDeserializerBuilder.of(paymentSpec)\n                                  .build();\n\n  try (var consumer = createConsumer()) {\n      consumer.subscribe(List.of(TOPIC));\n      while (true) {\n        ConsumerRecords\u003cString, JsObj\u003e records = consumer.poll(Duration.ofMillis(500));\n        System.out.println(\"Consumed \" + records.count() + \" records.\");\n        for (var record : records) {\n          Bytes kakfaBytes = record.value();\n          byte[] bytes = kakfaBytes.get();\n          JsObj obj = paymentDeserializer.deserialize(bytes);\n          System.out.printf(\"offset = %d, key = %s, value = %s%n\",\n                            record.offset(),\n                            record.key(),\n                            obj\n                           );\n        }\n      }\n    }\n\n```\n\n**Monitoring Serializers/Deserializers with JFR Events**\n\nAll the serializers and deserializers in this library support Java Flight Recorder (JFR) events. The\ndifferent events are:\n\n- `ConfluentSerializerEvent`\n- `ConfluentDeserializerEvent`\n- `SerializerEvent`\n- `DeserializerEvent`\n\nThere are four predefined formatters, which are functions that format the events into a string.\nConsider the following example to log some events using a JFR stream:\n\n```java\nRecordingStream rs = new RecordingStream();\n\nrs.setOrdered(true);\n\nString eventName = \"Confluent_Avro_Serializer_Event\";\n\nrs.onEvent(eventName,\n           recordedEvent -\u003e logger.info(ConfluentSerializerEventFormatter.apply(recordedEvent))\n          );\n\nrs.onEvent(eventName,\n           recordedEvent -\u003e logger.info(ConfluentDeserializerEventFormatter.apply(recordedEvent))\n          );\n\nrs.startAsync();\n\n```\n\nIf you are using [jio-test](), create a debugger in your test specifying the stream duration, and\nyou will see the events printed out on the console\n\n```java\n\n@RegisterExtension\nstatic Debugger debugger = Debugger.of(Duration.ofSeconds(5));\n\n```\n\n## \u003ca name=\"installation\"\u003e\u003ca/\u003e Installation\n\nTo include avro-spec in your project, add the corresponding dependency to your build tool based on\nyour Java version:\n\n```xml\n\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.github.imrafaelmerino\u003c/groupId\u003e\n  \u003cartifactId\u003eavro-spec\u003c/artifactId\u003e\n  \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n\n```\n\nRequires Java 21 or higher\n\n## \u003ca name=\"rit\"\u003e\u003ca/\u003e Running Integration Tests\n\nBefore executing the integration tests, ensure that Kafka is up and running, a topic is created, and\nan Avro schema is associated with the topic's values. To do this, follow the steps below, using the\nprovided [`docker-compose.yml`](src/test/resources/docker-compose.yml) file to start the Confluent platform:\n\n```shell\ncd src/test/resources\n\ndocker-compose up\n\nSCHEMA_REGISTRY_HOST=\"localhost\"\nSCHEMA_REGISTRY_PORT=\"8081\"\nREST_PROXY_PORT=\"8082\"\nREST_PROXY_HOST=\"localhost\"\nTOPIC_NAME=\"transactions\"\n\nCLUSTER_ID=$(curl -s -X GET -H \"Accept: application/json\" \"http://${REST_PROXY_HOST}:${REST_PROXY_PORT}/v3/clusters\" | jq -r '.data[0].cluster_id')\n\necho $CLUSTER_ID\n\ncurl -X POST \\\n-H \"Content-Type: application/json\" \\\n-H \"Accept: application/json\" \\\n--data '{\n    \"topic_name\": \"'${TOPIC_NAME}'\",\n    \"partitions_count\": 1,\n    \"replication_factor\": 1\n }' \\\n\"http://${REST_PROXY_HOST}:${REST_PROXY_PORT}/v3/clusters/${CLUSTER_ID}/topics\"\n\n\ncurl -X POST \\\n-H \"Content-Type: application/vnd.schemaregistry.v1+json\" \\\n--data @payment-schema.json \\\n\"http://${SCHEMA_REGISTRY_HOST}:${SCHEMA_REGISTRY_PORT}/subjects/${TOPIC_NAME}-value/versions\"\n\n```\n\nYou can verify that everything is working correctly by visiting the Control Center at\n[http://localhost:9021/clusters](http://localhost:9021/clusters).\n\nOnce Kafka is running, the topic is created, and the schema is registered, you're ready to run your\nintegration tests.\n\nTo run the tests, use one of the following Maven commands:\n\n- `mvn failsafe:integration-test` — To run integration tests only.\n- `mvn verify` — To run both unit and integration tests.\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimrafaelmerino%2Favro-spec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fimrafaelmerino%2Favro-spec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fimrafaelmerino%2Favro-spec/lists"}