{"id":27248838,"url":"https://github.com/bakdata/kafka-dead-letter-analyzer","last_synced_at":"2026-02-23T23:37:49.193Z","repository":{"id":38349293,"uuid":"468729146","full_name":"bakdata/kafka-dead-letter-analyzer","owner":"bakdata","description":"A Kafka Streams application that analyzes dead letters in your Kafka cluster","archived":false,"fork":false,"pushed_at":"2026-01-28T11:48:44.000Z","size":263,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":7,"default_branch":"main","last_synced_at":"2026-01-29T02:20:40.872Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bakdata.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-03-11T11:45:18.000Z","updated_at":"2026-01-28T11:48:46.000Z","dependencies_parsed_at":"2024-02-06T09:31:25.280Z","dependency_job_id":"21c13887-603c-4db6-b388-063c26def4cb","html_url":"https://github.com/bakdata/kafka-dead-letter-analyzer","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"purl":"pkg:github/bakdata/kafka-dead-letter-analyzer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-dead-letter-analyzer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-dead-letter-analyzer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-dead-letter-analyzer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-dead-letter-analyzer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bakdata","download_url":"https://codeload.github.com/bakdata/kafka-dead-letter-analyzer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bakdata%2Fkafka-dead-letter-analyzer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29760812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T21:02:23.375Z","status":"ssl_error","status_checked_at":"2026-02-23T20:58:31.539Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-10T23:48:15.627Z","updated_at":"2026-02-23T23:37:49.162Z","avatar_url":"https://github.com/bakdata.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://dev.azure.com/bakdata/public/_apis/build/status/bakdata.kafka-dead-letter-analyzer?repoName=bakdata%2Fkafka-dead-letter-analyzer\u0026branchName=main)](https://dev.azure.com/bakdata/public/_build/latest?definitionId=31\u0026repoName=bakdata%2Fkafka-dead-letter-analyzer\u0026branchName=main)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Adead-letter-analyzer\u0026metric=alert_status)](https://sonarcloud.io/summary/new_code?id=com.bakdata.kafka%3Adead-letter-analyzer)\n[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=com.bakdata.kafka%3Adead-letter-analyzer\u0026metric=coverage)](https://sonarcloud.io/summary/new_code?id=com.bakdata.kafka%3Adead-letter-analyzer)\n[![Maven](https://img.shields.io/maven-central/v/com.bakdata.kafka/dead-letter-analyzer.svg)](https://search.maven.org/search?q=g:com.bakdata.kafka%20AND%20a:dead-letter-analyzer\u0026core=gav)\n\n# kafka-dead-letter-analyzer\n\nA Kafka Streams application that analyzes dead letters in your Kafka cluster.\n\nDead letter analyzer supports three different types of dead letters in your Kafka cluster:\n\n- [Kafka Connect dead letters](https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/)\n- Kafka Streams dead letters in a similar format to Kafka Connect generated using [kafka-error-handling](https://github.com/bakdata/kafka-error-handling)\n- Kafka Streams dead letters in Apache Avro format generated using [kafka-error-handling](https://github.com/bakdata/kafka-error-handling)\n\nIt picks up all these dead letters and aggregates them by identifying different error types.\nAggregations are done by topic and are used to collect statistics for an error.\nThis includes the first, last, and number of occurrences of an error.\nThese statistics can then be stored in an external data system, such as a relational database or Elasticsearch,\nusing Kafka Connect or be directly explored in the respective Kafka topic.\nFurthermore, for each error type, one example record is selected in order to better understand what the error is about.\nFinally, all dead letters are sent to a topic in a standardized format along with its Kafka message context\n(topic, partition, offset, timestamp, key).\nThis data can also be stored in an external data system, such as Elasticsearch,\nto further explore the errors occurring in your data pipelines.\n\n## Getting started\n\nThe dead letter analyzer is available on [Docker Hub](https://hub.docker.com/repository/docker/bakdata/kafka-dead-letter-analyzer).\nYou can simply run it using docker:\n\n```\ndocker run bakdata/kafka-dead-letter-analyzer \\\n    --brokers localhost:9092 \\\n    --schema-registry-url http://localhost:8081 \\\n    --input-pattern .*-dead-letters \\\n    --output-topic dead-letters-all \\\n    --extra-output-topics stats=dead-letter-stats \\\n    --extra-output-topics examples=dead-letter-examples \\\n    --error-topic dead-letter-analyzer-dead-letters\n```\n\nThe analyzer then reads dead letters from any topic that matches the pattern `.*-dead-letters`.\nThese can be in any of the above-mentioned formats and the application automatically detects the serialization format.\nThe aggregated statistics are produced to the `dead-letter-stats` topic.\nExamples are forwarded to `dead-letter-examples` and all dead letters are sent to `dead-letters-all`.\nIf any processing error occurs in the analyzer itself, a dead letter is sent to `dead-letter-analyzer-dead-letters`.\nThese dead letters will be analyzed by the application as well.\n\nMessages sent to the examples and all topic are serialized using [kafka-large-message-serde](https://github.com/bakdata/kafka-large-message-serde).\n\n## Usage\n\nThis application is built using [streams-bootstrap](https://github.com/bakdata/streams-bootstrap) and can be configured as documented.\n\nWe suggest running your application on Kubernetes using Helm.\nYou can use the chart provided with [streams-bootstrap](https://github.com/bakdata/streams-bootstrap/tree/master/charts/streams-app).\n\n## Kafka Connect\n\nIf you want to store the analysis results in an external data system, such as Elasticsearch, you can do so using Kafka Connect.\nWe provide examples for each of the three output topics.\nHowever, they can be easily adapted to any data system supported by Kafka Connect.\n\n`dead-letters-all`:\n```json\n{\n    \"behavior.on.malformed.documents\": \"warn\",\n    \"connection.password\": \"password\",\n    \"connection.url\": \"http://localhost:9243\",\n    \"connection.username\": \"username\",\n    \"connector.class\": \"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector\",\n    \"errors.deadletterqueue.context.headers.enable\": \"true\",\n    \"errors.deadletterqueue.topic.name\": \"dead-letters-all-elasticsearch-sink-dead-letters\",\n    \"errors.tolerance\": \"all\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"key.ignore\": \"false\",\n    \"name\": \"dead-letters-all-elasticsearch-sink\",\n    \"tasks.max\": \"1\",\n    \"topics\": \"dead-letters-all\",\n    \"transforms\": \"changeTopic\",\n    \"transforms.changeTopic.timestamp.format\": \"yyyyMMdd\",\n    \"transforms.changeTopic.topic.format\": \"dead-letters-all-${timestamp}\",\n    \"transforms.changeTopic.type\": \"org.apache.kafka.connect.transforms.TimestampRouter\",\n    \"type.name\": \"_doc\",\n    \"value.converter\": \"com.bakdata.kafka.LargeMessageConverter\",\n    \"value.converter.large.message.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"value.converter.schema.registry.url\": \"http://localhost:8081\"\n}\n```\nThis connector partitions the dead letters by day so that you can set up Index Lifecycle Management for dropping old errors.\n\n`dead-letter-stats`:\n```json\n{\n    \"behavior.on.malformed.documents\": \"warn\",\n    \"connection.password\": \"password\",\n    \"connection.url\": \"http://localhost:9243\",\n    \"connection.username\": \"username\",\n    \"connector.class\": \"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector\",\n    \"errors.deadletterqueue.context.headers.enable\": \"true\",\n    \"errors.deadletterqueue.topic.name\": \"dead-letter-stats-elasticsearch-sink-dead-letters\",\n    \"errors.tolerance\": \"all\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"key.ignore\": \"false\",\n    \"name\": \"dead-letter-stats-elasticsearch-sink\",\n    \"tasks.max\": \"1\",\n    \"topics\": \"dead-letter-stats\",\n    \"transforms\": \"changeTopic\",\n    \"transforms.changeTopic.regex\": \".*\",\n    \"transforms.changeTopic.replacement\": \"dead-letter-stats\",\n    \"transforms.changeTopic.type\": \"org.apache.kafka.connect.transforms.RegexRouter\",\n    \"type.name\": \"_doc\",\n    \"value.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"value.converter.schema.registry.url\": \"http://localhost:8081\",\n    \"write.method\": \"upsert\"\n}\n```\n\n`dead-letter-examples`:\n```json\n{\n    \"behavior.on.malformed.documents\": \"warn\",\n    \"connection.password\": \"password\",\n    \"connection.url\": \"http://localhost:9243\",\n    \"connection.username\": \"username\",\n    \"connector.class\": \"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector\",\n    \"errors.deadletterqueue.context.headers.enable\": \"true\",\n    \"errors.deadletterqueue.topic.name\": \"dead-letter-examples-elasticsearch-sink-dead-letters\",\n    \"errors.tolerance\": \"all\",\n    \"key.converter\": \"org.apache.kafka.connect.storage.StringConverter\",\n    \"key.ignore\": \"false\",\n    \"name\": \"dead-letter-examples-elasticsearch-sink\",\n    \"tasks.max\": \"1\",\n    \"topics\": \"dead-letter-examples\",\n    \"transforms\": \"changeTopic\",\n    \"transforms.changeTopic.regex\": \".*\",\n    \"transforms.changeTopic.replacement\": \"dead-letter-stats\",\n    \"transforms.changeTopic.type\": \"org.apache.kafka.connect.transforms.RegexRouter\",\n    \"type.name\": \"_doc\",\n    \"value.converter\": \"com.bakdata.kafka.LargeMessageConverter\",\n    \"value.converter.large.message.converter\": \"io.confluent.connect.avro.AvroConverter\",\n    \"value.converter.schema.registry.url\": \"http://localhost:8081\",\n    \"write.method\": \"upsert\"\n}\n```\nThese two connectors use upsert semantics to store statistics alongside examples in the `dead-letter-stats` Elasticsearch index.\n\n## Development\n\nIf you want to contribute to this project, you can simply clone the repository and build it via Gradle.\nAll dependencies should be included in the Gradle files, there are no external prerequisites.\n\n```bash\n\u003e git clone git@github.com:bakdata/kafka-dead-letter-analyzer.git\n\u003e cd kafka-dead-letter-analyzer \u0026\u0026 ./gradlew build\n```\n\nPlease note, that we have [code styles](https://github.com/bakdata/bakdata-code-styles) for Java.\nThey are basically the Google style guide, with some small modifications.\n\n## Contributing\n\nWe are happy if you want to contribute to this project.\nIf you find any bugs or have suggestions for improvements, please open an issue.\nWe are also happy to accept your PRs.\nJust open an issue beforehand and let us know what you want to do and why.\n\n## License\nThis project is licensed under the MIT license.\nHave a look at the [LICENSE](https://github.com/bakdata/kafka-dead-letter-analyzer/blob/main/LICENSE) for more details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-dead-letter-analyzer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbakdata%2Fkafka-dead-letter-analyzer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbakdata%2Fkafka-dead-letter-analyzer/lists"}