{"id":21877622,"url":"https://github.com/rbiedrawa/spring-kafka-non-blocking-retries","last_synced_at":"2026-04-07T09:31:05.379Z","repository":{"id":162193129,"uuid":"377216825","full_name":"rbiedrawa/spring-kafka-non-blocking-retries","owner":"rbiedrawa","description":"This repository demonstrates how to handle Kafka Consumer exceptions in a non-blocking manner using retry topics and DLQ (Dead Letter Queue).","archived":false,"fork":false,"pushed_at":"2022-01-31T02:04:23.000Z","size":312,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-04T16:49:41.201Z","etag":null,"topics":["apache-kafka","dlq","docker","docker-compose","gradle","kafka","kafka-consumer","kowl","retries","spring-boot","spring-kafka"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rbiedrawa.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-15T15:49:52.000Z","updated_at":"2022-04-04T06:54:08.000Z","dependencies_parsed_at":null,"dependency_job_id":"526c1328-a1d0-4853-bb15-7f098e4deb1c","html_url":"https://github.com/rbiedrawa/spring-kafka-non-blocking-retries","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/rbiedrawa/spring-kafka-non-blocking-retries","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fspring-kafka-non-blocking-retries","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fspring-kafka-non-blocking-retries/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fspring-kafka-non-blocking-retries/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fspring-kafka-non-blocking-retries/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rbiedrawa","download_url":"https://codeload.github.com/rbiedrawa/spring-kafka-non-blocking-retries/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rbiedrawa%2Fspring-kafka-non-blocking-retries/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31507905,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-07T03:10:19.677Z","status":"ssl_error","status_checked_at":"2026-04-07T03:10:13.982Z","response_time":105,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-kafka","dlq","docker","docker-compose","gradle","kafka","kafka-consumer","kowl","retries","spring-boot","spring-kafka"],"created_at":"2024-11-28T08:09:48.016Z","updated_at":"2026-04-07T09:31:05.345Z","avatar_url":"https://github.com/rbiedrawa.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Non-Blocking Retries in Kafka Consumers (demo)\n\nThis repository demonstrates how to handle Kafka Consumer exceptions in a non-blocking manner using retry topics and DLQ (Dead Letter Queue).\n\n## Getting Started\n\n### Prerequisites\n\n* Java 11\n* Docker\n\n### Usage\n\n* Start docker-compose.\n  ```shell\n  docker-compose -f docker/docker-compose.yml up -d\n  ```\n\n* Start application.\n  ```shell\n  ./gradlew bootRun\n  ```\n\n* Run below command to produce random messages.\n  ```shell\n    (cd docker \u0026\u0026 docker-compose exec broker bash -c \"seq 2 | kafka-console-producer --request-required-acks 1 --broker-list localhost:29092 --topic my.topic.events \u0026\u0026 echo 'Produced 2 messages.'\")\n    \n    # \u003e\u003e\u003eProduced 2 messages.\n  ```\n\n* Investigate application logs.\n  ```shell\n  \n  # 2021-06-15 17:20:12.200 [-listener-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Received event 1 from topic my.topic.events\n  # 2021-06-15 17:20:12.710 [-listener-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-my-topic-listener-1, groupId=my-topic-listener] Seeking to offset 3 for partition my.topic.events-0\n  # 2021-06-15 17:20:12.711 [etry-1000-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-my-topic-listener-retry-1000-2, groupId=my-topic-listener-retry-1000] Seeking to offset 2 for partition my.topic.events-retry-1000-0\n  # 2021-06-15 17:20:12.711 [etry-1000-0-C-1] essageListenerContainer$ListenerConsumer : Seek to current after exception; nested exception is org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.kafka.listener.KafkaBackoffException: Partition 0 from topic my.topic.events-retry-1000 is not ready for consumption, backing off for approx. 490 millis.\n  # 2021-06-15 17:20:12.720 [-listener-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Received event 2 from topic my.topic.events\n  # 2021-06-15 17:20:13.223 [etry-1000-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Received event 1 from topic my.topic.events-retry-1000\n  # 2021-06-15 17:20:13.238 [etry-2000-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-my-topic-listener-retry-2000-3, groupId=my-topic-listener-retry-2000] Seeking to offset 2 for partition my.topic.events-retry-2000-0\n  # 2021-06-15 17:20:13.238 [etry-2000-0-C-1] essageListenerContainer$ListenerConsumer : Seek to current after exception; nested exception is org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.kafka.listener.KafkaBackoffException: Partition 0 from topic my.topic.events-retry-2000 is not ready for consumption, backing off for approx. 1986 millis.\n  # 2021-06-15 17:20:13.245 [etry-1000-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-my-topic-listener-retry-1000-2, groupId=my-topic-listener-retry-1000] Seeking to offset 3 for partition my.topic.events-retry-1000-0\n  # 2021-06-15 17:20:13.245 [etry-1000-0-C-1] essageListenerContainer$ListenerConsumer : Seek to current after exception; nested exception is org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.kafka.listener.KafkaBackoffException: Partition 0 from topic my.topic.events-retry-1000 is not ready for consumption, backing off for approx. 476 millis.\n  # 2021-06-15 17:20:13.867 [etry-1000-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Received event 2 from topic my.topic.events-retry-1000\n  # 2021-06-15 17:20:15.298 [etry-2000-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Received event 1 from topic my.topic.events-retry-2000\n  # 2021-06-15 17:20:15.810 [etry-2000-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-my-topic-listener-retry-2000-3, groupId=my-topic-listener-retry-2000] Seeking to offset 3 for partition my.topic.events-retry-2000-0\n  # 2021-06-15 17:20:15.812 [tener-dlt-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Event 1 handled by dlq topic: my.topic.events-dlt\n  # 2021-06-15 17:20:15.819 [etry-2000-0-C-1] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-my-topic-listener-retry-2000-3, groupId=my-topic-listener-retry-2000] Seeking to offset 3 for partition my.topic.events-retry-2000-0\n  # 2021-06-15 17:20:15.819 [etry-2000-0-C-1] essageListenerContainer$ListenerConsumer : Seek to current after exception; nested exception is org.springframework.kafka.listener.ListenerExecutionFailedException: Listener failed; nested exception is org.springframework.kafka.listener.KafkaBackoffException: Partition 0 from topic my.topic.events-retry-2000 is not ready for consumption, backing off for approx. 49 millis.\n  # 2021-06-15 17:20:16.326 [etry-2000-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Received event 2 from topic my.topic.events-retry-2000\n  # 2021-06-15 17:20:16.841 [tener-dlt-0-C-1] com.rbiedrawa.kafka.app.MyTopicListener  : Event 2 handled by dlq topic: my.topic.events-dlt\n  ```\n\n* Open your web browser and go to [Kowl Web UI page](http://localhost:8080/).\n    * Check how many consumer groups are active.\n      ![consumer-groups.png](./_docs/img/consumer-groups.png)\n\n    * Check kafka topics\n      ![topics.png](./_docs/img/topics.png)\n\n    * Investigate kafka message headers in retry and dlq topics\n      ![exception-message.png](./_docs/img/exception-message.png)\n\n## Important Endpoints\n\n| Name | Endpoint | \n| -------------:|:--------:|\n| `Kowl Web UI` | http://localhost:8080 |\n\n## References\n\n* [Spring Kafka docs - Non-Blocking Retries](https://docs.spring.io/spring-kafka/reference/html/#retry-topic)\n* [@RetryableTopic](https://docs.spring.io/spring-kafka/docs/current/api/org/springframework/kafka/annotation/RetryableTopic.html)\n* [Uber - Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka](https://eng.uber.com/reliable-reprocessing/)\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbiedrawa%2Fspring-kafka-non-blocking-retries","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frbiedrawa%2Fspring-kafka-non-blocking-retries","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frbiedrawa%2Fspring-kafka-non-blocking-retries/lists"}