{"id":27248852,"url":"https://github.com/bakdata/kafka-error-handling","last_synced_at":"2025-04-10T23:48:18.450Z","repository":{"id":38257363,"uuid":"245396430","full_name":"bakdata/kafka-error-handling","owner":"bakdata","description":"A library for error handling in Kafka Streams.","archived":false,"fork":false,"pushed_at":"2025-03-20T12:03:38.000Z","size":396,"stargazers_count":18,"open_issues_count":0,"forks_count":3,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-10T23:48:11.197Z","etag":null,"topics":["java","kafka-streams"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-06T10:55:41.000Z","updated_at":"2025-03-20T12:03:42.000Z","dependencies_parsed_at":"2023-10-16T19:19:22.731Z","dependency_job_id":"8ce4db41-1bf6-4245-a615-60a0f97d421e","html_url":"https://github.com/bakdata/kafka-error-handling","commit_stats":null,"previous_names":[],"tags_count":20,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-error-handling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-error-handling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-error-handling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-error-handling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/kafka-error-handling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248317727,"owners_count":21083528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","kafka-streams"],"created_at":"2025-04-10T23:48:17.828Z","updated_at":"2025-04-10T23:48:18.441Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.kafka-error-handling?branchName=master)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=23\u0026branchName=master)\n[![Sonarcloud status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Aerror-handling\u0026metric=alert_status)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Aerror-handling)\n[![Code coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Aerror-handling\u0026metric=coverage)](https://sonarcloud.io/dashboard?id=com.bakdata.kafka%3Aerror-handling)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/error-handling-core.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:error-handling-core\u0026core=gav)\n\n# Kafka error handling\nLibraries for error handling in [Kafka Streams](https://kafka.apache.org/documentation/streams/).\n\n## Getting Started\n\nYou can add Kafka error handling via Maven Central.\nDepending on how you want to store the dead letters in Kafka, you can use the Avro or Protobuf converter. \n\n#### Gradle\n```gradle\n// for Avro dead letters\ncompile group: 'com.bakdata.kafka', name: 'error-handling-avro', version: '1.3.0'\n// or, for Protobuf dead letters\ncompile group: 'com.bakdata.kafka', name: 'error-handling-proto', version: '1.3.0'\n// or, for custom dead letters\ncompile group: 'com.bakdata.kafka', name: 'error-handling-core', version: '1.3.0'\n```\n\n#### Maven\n```xml\n\u003c!-- for Avro dead letters --\u003e\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003eerror-handling-avro\u003c/artifactId\u003e\n    \u003cversion\u003e1.3.0\u003c/version\u003e\n\u003c/dependency\u003e\n\n\u003c!-- or, for Protobuf dead letters --\u003e\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003eerror-handling-proto\u003c/artifactId\u003e\n    \u003cversion\u003e1.3.0\u003c/version\u003e\n\u003c/dependency\u003e\n\n\u003c!-- or, for custom dead letters --\u003e\n\u003cdependency\u003e\n\u003cgroupId\u003ecom.bakdata.kafka\u003c/groupId\u003e\n\u003cartifactId\u003eerror-handling-core\u003c/artifactId\u003e\n\u003cversion\u003e1.3.0\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor other build tools or versions, refer to the [latest version in MvnRepository](https://mvnrepository.com/artifact/com.bakdata.kafka/error-handling/latest).\n\n### Usage\n\nIf you use Kafka Streams to process your data, you will sooner or later get to the point where processing of a message throws an exception.\nIn case your streams application is configured to process every message at least once, which is the case most times,\nyour application will crash upon encountering an error and retry processing the erroneous message.\nIf the error was just temporary, processing will continue as if nothing has happened.\nHowever, if the error is related to the specific message, then your streams application will be stuck processing the record.\nFor such cases we developed three solutions that help handling errors in your Kafka Streams application.\n\nConsider the following Topology:\n\n```java\nfinal KeyValueMapper\u003cInteger, String, KeyValue\u003cDouble, Long\u003e\u003e mapper = …\nfinal KStream\u003cInteger, String\u003e input =\n       builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.String()));\n\nfinal KStream\u003cDouble, Long\u003e mapped = input.map(mapper);\nmapped.to(OUTPUT_TOPIC, Produced.with(Serdes.Double(), Serdes.Long()));\n```\n\nYou can simply add a dead letter queue to your topology with our libraries:\n\n```java\nfinal KeyValueMapper\u003cInteger, String, KeyValue\u003cDouble, Long\u003e\u003e mapper = …\nfinal KStream\u003cInteger, String\u003e input =\n       builder.stream(INPUT_TOPIC, Consumed.with(Serdes.Integer(), Serdes.String()));\n\nfinal KStream\u003cDouble, ProcessedKeyValue\u003cInteger, String, Long\u003e\u003e mappedWithErrors =\n       input.map(captureErrors(mapper));\nmappedWithErrors.flatMap(ProcessedKeyValue::getErrors)\n       .processValues(AvroDeadLetterConverter.asProcessor(\"A good description where the pipeline broke\"))\n       .to(ERROR_TOPIC);\n\nfinal KStream\u003cDouble, Long\u003e mapped = mappedWithErrors.flatMapValues(ProcessedKeyValue::getValues);\nmapped.to(OUTPUT_TOPIC, Produced.with(Serdes.Double(), Serdes.Long()));\n```\n\nSuccessfully processed messages are sent to the output topic as before.\nHowever, errors are sent to a specific error topic.\nThis error topic contains dead letters describing the input value, error message and stack trace of any error that is raised in that part of your topology.\n\nThe example uses the `AvroDeadLetterConverter` from `error-handling-avro`.\nAnalogously, `error-handling-proto` implements a `ProtoDeadLetterConverter`.\nA custom `DeadLetterConverter` can be passed to `DeadLetterProcessor.create`.\n\n## Development\n\nIf you want to contribute to this project, you can simply clone the repository and build it via Gradle.\nAll dependencies should be included in the Gradle files, there are no external prerequisites.\n\n```bash\n\u003e git clone git@github.com:bakdata/kafka-error-handling.git\n\u003e cd kafka-error-handling \u0026\u0026 ./gradlew build\n```\n\nPlease note, that we have [code styles](https://github.com/bakdata/bakdata-code-styles) for Java.\nThey are basically the Google style guide, with some small modifications.\n\n## Contributing\n\nWe are happy if you want to contribute to this project.\nIf you find any bugs or have suggestions for improvements, please open an issue.\nWe are also happy to accept your PRs.\nJust open an issue beforehand and let us know what you want to do and why.\n\n## License\nThis project is licensed under the MIT license.\nHave a look at the [LICENSE](https://github.com/bakdata/kafka-error-handling/blob/master/LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-error-handling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fkafka-error-handling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-error-handling/lists"}