{"id":27248844,"url":"https://github.com/bakdata/kafka-brute-force-serde","last_synced_at":"2025-04-10T23:48:16.777Z","repository":{"id":39608851,"uuid":"468655958","full_name":"bakdata/kafka-brute-force-serde","owner":"bakdata","description":"A Kafka SerDe to deserialize messages of an unknown serialization format","archived":false,"fork":false,"pushed_at":"2025-03-21T06:46:54.000Z","size":290,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-10T23:48:11.299Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-11T07:45:36.000Z","updated_at":"2025-03-21T06:46:58.000Z","dependencies_parsed_at":"2023-11-10T14:30:57.914Z","dependency_job_id":"926dd4dd-1c9e-4999-bf21-5e65ce635c9c","html_url":"https://github.com/bakdata/kafka-brute-force-serde","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-brute-force-serde","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-brute-force-serde/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-brute-force-serde/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-brute-force-serde/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/kafka-brute-force-serde/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317727,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-10T23:48:16.283Z","updated_at":"2025-04-10T23:48:16.772Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.kafka-brute-force-serde?repoName=bakdata%2Fkafka-brute-force-serde\u0026branchName=main)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=30\u0026repoName=bakdata%2Fkafka-brute-force-serde\u0026branchName=main)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Abrute-force\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=com.bakdata.kafka%3Abrute-force)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Abrute-force\u0026metric=coverage)](https://sonarcloud.io/summary/new_code?id=com.bakdata.kafka%3Abrute-force)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/brute-force-serde.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:brute-force-serde\u0026core=gav)\n\n# kafka-brute-force-serde\nA Kafka SerDe to deserialize messages of an unknown serialization format\n\n## Getting Started\n\n### Serde\n\nYou can add kafka-brute-force-serde via Maven Central.\n\n#### Gradle\n```gradle\nimplementation group: 'com.bakdata.kafka', name: 'brute-force-serde', version: '1.0.0'\n```\n\n#### Maven\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003ebrute-force-serde\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/brute-force-serde/latest).\n\nMake sure to also add [Confluent Maven Repository](http://packages.confluent.io/maven/) to your build file.\n\n#### Usage\n\nYou can use it from your Kafka Streams application like any other Serde\n\n```java\nfinal Serde\u003cObject\u003e serde = new BruteForceSerde\u003c\u003e();\nserde.configure(Map.of(), false);\n```\n\nBy default, the Serde attempts to deserialize messages using the following Serdes in this order:\n- SpecificAvroSerde \n- GenericAvroSerde \n- StringSerde\n- ByteArraySerde\n\nFor each serialization format, BruteForceSerde first attempts deserialization using [Kafka Large Message Serde](https://github.com/bakdata/kafka-large-message-serde/)\nand then uses the standard format.\n\nYou can find a list of [all brute force configurations below](#configuration).\nAll other configuration properties are also delegated to the nested Serdes used by BruteForceSerde.\n\n### Kafka Connect\n\nThis serde also comes with support for Kafka Connect.\nYou can add kafka-brute-force-connect via Maven Central.\n\n#### Gradle\n```gradle\nimplementation group: 'com.bakdata.kafka', name: 'brute-force-connect', version: '1.0.0'\n```\n\n#### Maven\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003ebrute-force-connect\u003c/artifactId\u003e\n    \u003cversion\u003e1.0.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/brute-force-connect/latest).\n\n#### Usage\n\nTo use it with your Kafka Connect connectors, just configure your converter as `com.bakdata.kafka.BruteForceConverter`.\n\nBy default, the converter attempts to deserialize messages using the following converters in this order:\n- AvroConverter\n- StringConverter\n- ByteArrayConverter\n\nFor each serialization format, BruteForceSerde first attempts deserialization using [Kafka Large Message Converter](https://github.com/bakdata/kafka-large-message-serde/)\nand then uses the standard format.\n\nYou can find a list of [all brute force configurations below](#configuration).\nAll configuration properties are also delegated to the nested Converters used by BruteForceConverter.\n\nFor general guidance on how to configure Kafka Connect converters, please have a look at the [official documentation](https://docs.confluent.io/home/connect/configuring.html).\n\n\n## Configuration\n\n``brute.force.large.message.enabled``\nFlag for enabling support for large-message-serde.\n\n* Type: boolean\n* Default: true\n* Importance: low\n\n``brute.force.ignore.no.match``\nIf set, the deserialization won't fail and instead keep the data as a byte array. This is equivalent to including ByteArray conversion in the corresponding conversion list.\n\n* Type: boolean\n* Default: true\n* Importance: low\n\n### Serde\n\n``brute.force.serdes``\nA comma separated list of SerDes that should be tried.\n\n* Type: list\n* Default: `io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde,io.confluent.kafka.streams.serdes.avro.GenericAvroSerde,org.apache.kafka.common.serialization.Serdes$StringSerde,org.apache.kafka.common.serialization.Serdes$ByteArraySerde`\n* Importance: medium\n\n\n### Kafka Connect\n\n``brute.force.converters``\nA comma separated list of converters that should be tried.\n\n* Type: list\n* Default: `io.confluent.connect.avro.AvroConverter,org.apache.kafka.connect.storage.StringConverter,org.apache.kafka.connect.converters.ByteArrayConverter`\n* Importance: medium\n\n\n## Development\n\nIf you want to contribute to this project, you can simply clone the repository and build it via Gradle.\nAll dependencies should be included in the Gradle files, there are no external prerequisites.\n\n```bash\n\u003e git clone git@github.com:bakdata/kafka-brute-force-serde.git\n\u003e cd kafka-brute-force-serde \u0026\u0026 ./gradlew build\n```\n\nPlease note, that we have [code styles](https://github.com/bakdata/bakdata-code-styles) for Java.\nThey are basically the Google style guide, with some small modifications.\n\n## Contributing\n\nWe are happy if you want to contribute to this project.\nIf you find any bugs or have suggestions for improvements, please open an issue.\nWe are also happy to accept your PRs.\nJust open an issue beforehand and let us know what you want to do and why.\n\n## License\nThis project is licensed under the MIT license.\nHave a look at the [LICENSE](https://github.com/bakdata/kafka-brute-force-serde/blob/main/LICENSE) for more details.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-brute-force-serde","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fkafka-brute-force-serde","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-brute-force-serde/lists"}