{"id":13566043,"url":"https://github.com/saubury/kafka-serialization","last_synced_at":"2025-04-03T23:31:01.202Z","repository":{"id":41107371,"uuid":"260078851","full_name":"saubury/kafka-serialization","owner":"saubury","description":"Experiments and demonstrations of AVRO, Protobuf serialisation","archived":false,"fork":false,"pushed_at":"2022-12-08T09:44:29.000Z","size":272,"stargazers_count":60,"open_issues_count":3,"forks_count":14,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-04T20:41:36.366Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/saubury.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-30T00:45:32.000Z","updated_at":"2024-10-30T00:48:36.000Z","dependencies_parsed_at":"2023-01-25T05:15:47.541Z","dependency_job_id":null,"html_url":"https://github.com/saubury/kafka-serialization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saubury%2Fkafka-serialization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saubury%2Fkafka-serialization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saubury%2Fkafka-serialization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/saubury%2Fkafka-serialization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/saubury","download_url":"https://codeload.github.com/saubury/kafka-serialization/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247097662,"owners_count":20883122,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:02:00.783Z","updated_at":"2025-04-03T23:30:59.907Z","avatar_url":"https://github.com/saubury.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"Experiments with Kafka serialization schemes\n\n# Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema  \n\nApache Avro was has been the defacto Kafka serialization mechanism for a long time.   Confluent [just updated](https://www.confluent.io/blog/confluent-platform-now-supports-protobuf-json-schema-custom-formats/) their Kafka streaming platform with additioinal support for serializing data with Protocol buffers (or _protobuf_) and JSON Schema serialization.  \n\n![alt splash](./docs/landing.png )\n\n\nProtobuf is especially cool, and offers up some neat opportunities beyond what was possible in Avro.  The inclusion of Protobuf and JSON Schema applies at producer and consumer libraries, schema registry, Kafka connect, ksqlDB along with Control Center.\n\n\n## Do I care about serializing structured data?\n\nSo why bother with serializing structured data?  Let's start with an example data string ... `cookie,50,null`.  What does this data mean? Is _cookie_ a name, a place or something to eat?  And what about _50_ - is this an age, a temperature or something else?\n\nIf you were using a database (such as Postgres or Oracle) to store your data you would create a table definition (with nicely named columns and appropriate data types). The same is true for your streaming platform - you really should pick data format for serializing structured data. Bonus points for and being consistent across your data platform!\n\n\n\n\nUntil recently your choices for serializing structured data within Kafka were limited. You had \"bad\" choices (such as free text or CSV) or the \"better\" choice of using Apache Avro. Avro is an open source data serialization system which marshals your data (and it's appropriate schema) to a efficient binary format.  One of the core features of Avro is the ability to define a schema for our data. So our data `cookie,50,null` would be associated with a _snack_ Avro schema like this\n\n```json\n{\n  \"type\": \"record\",\n  \"name\": \"snacks\",\n  \"fields\": [\n      {\"name\": \"name\",  \"type\": \"string\" }\n    , {\"name\": \"calories\", \"type\": \"float\" }\n    , {\"name\": \"colour\", \"type\": \"string\", \"default\": null}\n  ]\n}\n```\n\nHere we can see our data `cookie,50,null` is snack data (the most important type of data). We can see _cookie_ is a _string_ representing the name of the snake. Our schema offers us a lot of flexibility (our schema can evolve over time) plus ensures data integrity (for example, ensuring calories are integers ).  \n\n\nAlthough most of Apache Kafka users use Apache Avro to define contracts for their messages, it's always been a bit of a \"Java thing\".  Classes automatically generated by the Apache Avro compiler favor JVM developers. You can certainly use AVRO in pretty much any language, however, Google Protocol Buffer (protobuf) is very popular for serializing, deserializing and validating data in other languages (Python, Rust, Ruby, Go).\n\n\n\n# AVRO, Protobuf, JSON Schema use with Kafka\n\nThis _isn't_ a blog on the \"best\" serialization strategy. However, let's get familiar with how we can use new choices for serializing structured data\n\nOur initial set of yummy data looks like this\n\n```json\n{\"name\": \"cookie\", \"calories\": 500, \"colour\": \"brown\"}\n{\"name\": \"cake\", \"calories\": 260, \"colour\": \"white\"}\n{\"name\": \"timtam\", \"calories\": 80, \"colour\": \"chocolate\"}\n```\n\n## AVRO serialization \nLet's remind ourselves how to encode our snacks using AVRO serialization. We'll use the include command line tool `kafka-avro-console-producer` as a Kafka producer which can perform serialization (with a schema provided as a command line parameter).  A _producer_ is something that _writes_ data into a Kafka broker.\n\n```bash\nkafka-avro-console-producer  --broker-list localhost:9092 --topic SNACKS_AVRO --property value.schema='\n{\n  \"type\": \"record\",\n  \"name\": \"myrecord\",\n  \"fields\": [\n      {\"name\": \"name\",  \"type\": \"string\" }\n    , {\"name\": \"calories\", \"type\": \"float\" }\n    , {\"name\": \"colour\", \"type\": \"string\" }\n  ]\n}' \u003c snacks.txt\n```\n\nAnd to _read_ the data, we can use the `kafka-avro-console-consumer` command line application to act as kafka consumer to read and de-serializing our AVRO data\n\n```bash\nkafka-avro-console-consumer --bootstrap-server localhost:9092 --topic SNACKS_AVRO --from-beginning \n\n{\"name\":\"cookie\",\"calories\":500.0,\"colour\":\"brown\"}\n{\"name\":\"cake\",\"calories\":260.0,\"colour\":\"white\"}\n{\"name\":\"timtam\",\"calories\":80.0,\"colour\":\"chocolate\"}\n```\n\n## Protocol Buffers (Protobuf) serialization \n\nThis time we'll use _protobuf_ serialization with the new `kafka-protobuf-console-producer` kafka producer. The concept is similar to to approach we took with AVRO, however this time our Kafka producer will can perform protobuf serialization.  Note the protobuf schema is provided as a command line parameter.\n\n```bash\nkafka-protobuf-console-producer --broker-list localhost:9092 --topic SNACKS_PROTO --property value.schema='\nmessage Snack {\n    required string name = 1;\n    required int64 calories = 2;\n    optional string colour = 3;\n}' \u003c snacks.txt\n```\n\nAnd to read the data, we can use the `kafka-protobuf-console-consumer` kafka consumer for de-serializing our protobuf data\n\n```bash\nkafka-protobuf-console-consumer --bootstrap-server localhost:9092 --topic SNACKS_PROTO --from-beginning \n\n{\"name\":\"cookie\",\"calories\":\"500\",\"colour\":\"brown\"}\n{\"name\":\"cake\",\"calories\":\"260\",\"colour\":\"white\"}\n{\"name\":\"timtam\",\"calories\":\"80\",\"colour\":\"chocolate\"}\n```\n\n\n## JSON Schema serialization \n\nFinally we'll use JSON Schema serialization with the new `kafka-json-schema-console-producer` kafka producer.  Note the json-schema schema is provided as a command line parameter.\n\n```bash\nkafka-json-schema-console-producer --broker-list localhost:9092 --topic SNACKS_JSONSCHEMA --property value.schema='\n{\n  \"definitions\" : {\n    \"record:myrecord\" : {\n      \"type\" : \"object\",\n      \"required\" : [ \"name\", \"calories\" ],\n      \"additionalProperties\" : false,\n      \"properties\" : {\n        \"name\" : {\"type\" : \"string\"},\n        \"calories\" : {\"type\" : \"number\"},\n        \"colour\" : {\"type\" : \"string\"}\n      }\n    }\n  },\n  \"$ref\" : \"#/definitions/record:myrecord\"\n}' \u003c snacks.txt\n```\n\nAnd to read the data, we can use the `kafka-json-schema-console-consumer` kafka consumer for de-serializing our json-schema data\n\n```bash\nkafka-json-schema-console-consumer --bootstrap-server localhost:9092 --topic SNACKS_JSONSCHEMA --from-beginning \n\n{\"name\":\"cookie\",\"calories\":\"500\",\"colour\":\"brown\"}\n{\"name\":\"cake\",\"calories\":\"260\",\"colour\":\"white\"}\n{\"name\":\"timtam\",\"calories\":\"80\",\"colour\":\"chocolate\"}\n```\n\n# Protobuf with the Confluent Schema Registry\n\nYou may have wondered where the schemas actually went in the above examples?  The Confluent Schema Registry has been diligently storing these schemas (as part of the serialization process when using kafka-blah-console-producer).  That is, both the schema name (eg., _SNACKS_PROTO-value_), schema content, version and style (protobuf, Avro) have all been stored.  We can peak at the stored schema with `curl`. For example, to explore the recently used protobuf schema for our snacks\n\n```bash\ncurl -s -X GET http://localhost:8081/subjects/SNACKS_PROTO-value/versions/1 \n```\n\nWhich responds the this snack schema (yummy)\n\n```json\n{\n  \"subject\": \"SNACKS_PROTO-value\",\n  \"version\": 1,\n  \"id\": 6,\n  \"schemaType\": \"PROTOBUF\",\n  \"schema\": \"\\nmessage Snack {\\n  required string name = 1;\\n  required int64 calories = 2;\\n  required string colour = 3;\\n}\\n\"\n}\n```\n\n\n# What you can do with Protobuf and can't do with Avro?\n\nLet's have a look at a more complex modeling example to illustrate some of the possibilities with Protobuf schemas.  Imagine we want to model a meal and describe the ingredients within the meal. We can use a protobuf schema to describe a meal such as a _taco_ is composed of _beef filling_ and _cheese topping_.\n\nOur data for a taco and fish-and-chips meal could look like this\n\n```json\n{\n  \"name\": \"tacos\",\n  \"item\": [\n    {\n      \"item_name\": \"beef\",\n      \"type\": \"FILLING\"\n    },\n    {\n      \"item_name\": \"cheese\",\n      \"type\": \"TOPPING\"\n    }\n  ]\n}, {\n  \"name\": \"fish and chips\",\n  \"alternate_name\": \"fish-n chips\",\n  \"item\": []\n}\n```\n\nAn example protobuf schema to represent our meals would look like this\n\n```javascript\nmessage Meal {\n  required string name = 1;\n  optional string alternate_name = 2;\n\n  enum FoodType {\n    INGREDIENT = 0;\n    FILLING = 1;\n    TOPPING = 2;\n  }\n\n  message MealItems {\n    required string item_name = 1;\n    optional FoodType type = 2 [default = INGREDIENT];\n  }\n\n  repeated MealItems item = 4;\n}\n```\n\n\nTo try this modeling with protobuf in Kafka\n\n\n```bash\nkafka-protobuf-console-producer --broker-list localhost:9092 --topic MEALS_PROTO --property value.schema='\nmessage Meal {\n  required string name = 1;\n  optional string alternate_name = 2;\n\n  enum FoodType {\n    INGREDIENT = 0;\n    FILLING = 1;\n    TOPPING = 2;\n  }\n\n  message MealItems {\n    required string item_name = 1;\n    optional FoodType type = 2 [default = INGREDIENT];\n  }\n\n  repeated MealItems item = 4;\n}' \u003c meals.txt\n```\n\nWhich gives you an idea of how flexible data representation can be using protobuf in Kafka.  But what happens if we need to make changes to these schemas?\n\n\n# Schema Evolution with Protobuf \nAs the saying goes - _the only constant is change_. Any good data platform needs to accommodate changes - such as additions or changes to a schema.  Supporting schema evolution is a fundamental requirement for a streaming platform, so our serialization mechanism also needs to support schema changes (or evolution). Protobuf and Avro both offer mechanisms for updating schema without breaking downstream consumers - which could still be using a previous schema version.\n\n## Adding drinks to our meals\n\nTacos and pizza's sound great - but let's have something to drink with our meal!  We can now add some additional attributes to our schema to include meals.  This is sometimes called _schema evolution_.  Note we'll continue to use the existing `MEALS_PROTO` topic.\n\nA new data payload (with the inclusion of _beer_)\n\n```json\n{\n  \"name\": \"pizza\",\n  \"drink\": [\n    {\n      \"drink_name\": \"beer\",\n      \"type\": \"ALCOHOLIC\"\n    }\n  ]\n}\n```\n\n```bash\nkafka-protobuf-console-producer --broker-list localhost:9092 --topic MEALS_PROTO --property value.schema='\nmessage Meal {\n  required string name = 1;\n  optional string alternate_name = 2;\n\n  enum FoodType {\n    INGREDIENT = 0;\n    FILLING = 1;\n    TOPPING = 2;\n  }\n\n  enum DrinkType {\n    BUBBLY = 0;\n    ALCOHOLIC = 1;\n  }\n\n  message MealItems {\n    required string item_name = 1;\n    optional FoodType type = 2 [default = INGREDIENT];\n  }\n\n  message DrinkItems {\n    required string drink_name = 1;\n    optional DrinkType type = 2 ;\n  }\n\n  repeated MealItems item = 4;\n  repeated DrinkItems drink = 5;\n}' \u003c meals-2.txt\n```\n\n## Visualizing schema difference with Confluent Control Center\n\nOne nice inclusion with the _Confluent Control Center_ (the Web GUI included in the Confluent platform) is the ability to look at schemas, and the _differences between schemas_.  For example, we can see version 1 and version 2 of the `MEALS_PROTO-value` schema\n\n![alt schema_control_center](./docs/schema_control_center.png \"Schema difference with Confluent Control Center\")\n\n\n\n# Application Binding - Protobuf classes with Python\nLet us now build an application demonstrating protobuf classes.  To generate protobuf classes you must first install the protobuf compiler `protoc`.  See the [protocol buffer docs](https://developers.google.com/protocol-buffers/docs/pythontutorial) for instructions on installing and using protoc.\n\n\n### Python compile schema\n```bash\nprotoc -I=. --python_out=. ./meal.proto\n```\n\nThis will create the `meal_pb2.py` Python class file.  You can now build protobuf classes and produce into Kafka with code like this\n\n```python\nimport meal_pb2\n \nmybeer = Meal.DrinkItems(drink_name=\"beer\")\nmywine = Meal.DrinkItems(drink_name=\"wine\")\nmeal =   Meal(name='pizza', drink=[mybeer,mywine])\n\nproducer.produce(topic='MEAL_PROTO', value=meal)\n```\n\nHave a look at [producer-protobuf.py](https://github.com/saubury/kafka-serialization/blob/master/producer-protobuf.py) for a complete example of protobuf Kafka producer in Python.\n\n### Setup Python virtual environment \n\n```bash\nvirtualenv myenv\n. ./myenv/bin/activate\npip install -r requirements.txt\n```\n\n\n# Conclusion\n\nKafka continues to grow in capabilities, and having the options of AVRO, Protobuf, JSON Schema use within the Confluent Platform gives even more opportunities to build cool streaming applications\nThe code for these examples available at\nhttps://github.com/saubury/kafka-serialization\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaubury%2Fkafka-serialization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsaubury%2Fkafka-serialization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsaubury%2Fkafka-serialization/lists"}