{"id":26474301,"url":"https://github.com/conduktor/my_custom_deserializers","last_synced_at":"2025-03-19T22:52:11.962Z","repository":{"id":38327431,"uuid":"429389886","full_name":"conduktor/my_custom_deserializers","owner":"conduktor","description":"Kafka deserializer examples","archived":false,"fork":false,"pushed_at":"2025-01-28T15:41:14.000Z","size":95,"stargazers_count":2,"open_issues_count":7,"forks_count":1,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-01-28T16:22:03.647Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/conduktor.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-18T10:40:36.000Z","updated_at":"2025-01-28T15:39:27.000Z","dependencies_parsed_at":"2024-04-04T10:43:13.967Z","dependency_job_id":null,"html_url":"https://github.com/conduktor/my_custom_deserializers","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fmy_custom_deserializers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fmy_custom_deserializers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fmy_custom_deserializers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/conduktor%2Fmy_custom_deserializers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/conduktor","download_url":"https://codeload.github.com/conduktor/my_custom_deserializers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244520144,"owners_count":20465627,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-19T22:52:11.267Z","updated_at":"2025-03-19T22:52:11.950Z","avatar_url":"https://github.com/conduktor.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Conduktor custom deserializers examples\n\nThis repository contains examples of custom deserializers usable in Conduktor with plugin mechanism.\n\nYou can download the latest jar containing these deserializers [my_custom_deserializers_2.13-2.3.0.jar](https://github.com/conduktor/my_custom_deserializers/packages/1105212)\n\nTo learn how to use the \"custom deserializer\" feature see [Conduktor documentation](https://docs.conduktor.io/features/consuming-data/custom-deserializers)\n\n## Simple example \n\n`io.example.conduktor.custom.deserializers.MyCustomDeserializer`\n\n[visible here](./src/main/scala/io/example/conduktor/custom/deserializers/MyCustomDeserializer.scala)\n\n\nThis deserializer transforms the data (bytes) it receives from Kafka into a `String` (text),     \nthen sees if it matches then following format:\n```\n-- this is the serialized data\n```\n- If the message received from Kafka effectively starts with a `--\u003cspace\u003e` characters sequence then followed by some text, \nit creates a new instance of a data structure named `MyMessage`, that contains only one field named `value` and is of type `String`, as following:     \n    ```scala\n    MyMessage(value = \"this is the serialized data\")\n    ```\n\n    In Conduktor, this data structure will be interpreted and displayed as JSON:     \n    ```json\n    { \"value\": \"this is the serialized data\" }\n    ```\n  \n- If the message received from Kafka doesn't match the expected format, then the deserializer fails with an error message:\n  ```\n  Invalid format received for message: \u003cthe received message\u003e\n  ```\n\nThis simple example is here to demonstrate 2 things:\n  - What's happening when a custom deserializer fails to deserialize a message.\n  - Give a simplified example of \"deserialization\" (the message has to respect of certain format so that the deserializer can extract the data)\n\n\nTo see simple example around constants, jump [here](./doc/details.md)\n\n## Protobuf example\n`io.example.conduktor.custom.deserializers.MyCustomProtobufDeserializer`\n\n[located here](./src/main/scala/io/example/conduktor/custom/deserializers/MyCustomProtobufDeserializer.scala)\n\n\nThis example allow to deserialize a protobuf payload corresponding to this schema : \n\n```\nmessage Person {\n  required string name = 1;\n  required int32 id = 2;\n  optional string email = 3;\n\n  enum PhoneType {\n    MOBILE = 0;\n    HOME = 1;\n    WORK = 2;\n  }\n\n  message PhoneNumber {\n    required string number = 1;\n    optional PhoneType type = 2 [default = HOME];\n  }\n\n  repeated PhoneNumber phones = 4;\n}\n```\n\n## Configurable example\n`io.example.conduktor.custom.deserializers.MyConfigurableDeserializer`\n\n[located here](./src/main/scala/io/example/conduktor/custom/deserializers/MyConfigurableDeserializer.scala)\n\nThis example allow to show deserializer configuration to change it's behavior. \nTo configure the behabor, the Deserializer check for a `output` property in it's configuration. \n\n### Passthrough mode : \nWith configuration :\n```properties\noutput=passthrough\n```\nThe data on record are not de coded and returned as-is in bytes array form.\n\n### Config mode :\nWith configuration :\n```properties\noutput=config\n```\nThe configuration is returned on each record deserialization. \nFor example with configuration \n```properties\noutput=config\nother.property=some value\n```\nWill always return JSON like \n```json\n{\n  \"output\": \"config\",\n  \"other.property\": \"some value\"\n}\n```\n\n### Constant mode :\n\nWith configuration output defined to something else other than `config` or `passthrough` and not empty like:\n```properties\noutput=some constant output\n```\nThe Deserializer will always return String value like \n```json\n\"some constant output\"\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduktor%2Fmy_custom_deserializers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconduktor%2Fmy_custom_deserializers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconduktor%2Fmy_custom_deserializers/lists"}