{"id":29708778,"url":"https://github.com/michelin/kafka-retryable-consumer","last_synced_at":"2025-07-23T19:06:35.566Z","repository":{"id":304119204,"uuid":"890861588","full_name":"michelin/kafka-retryable-consumer","owner":"michelin","description":"Ease and secure your Kafka consumers with error\u0026retry capabilities","archived":false,"fork":false,"pushed_at":"2025-07-21T07:56:23.000Z","size":173,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-21T08:39:47.486Z","etag":null,"topics":["java","kafka","kafka-consumer"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/michelin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-11-19T10:06:09.000Z","updated_at":"2025-07-17T13:34:39.000Z","dependencies_parsed_at":"2025-07-11T10:31:42.562Z","dependency_job_id":"24f9c8a3-f243-4b50-b25f-1c0275da2231","html_url":"https://github.com/michelin/kafka-retryable-consumer","commit_stats":null,"previous_names":["michelin/kafka-retryable-consumer"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/michelin/kafka-retryable-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelin%2Fkafka-retryable-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelin%2Fkafka-retryable-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelin%2Fkafka-retryable-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelin%2Fkafka-retryable-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/michelin","download_url":"https://codeload.github.com/michelin/kafka-retryable-consumer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/michelin%2Fkafka-retryable-consumer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266737742,"owners_count":23976392,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","kafka","kafka-consumer"],"created_at":"2025-07-23T19:06:34.907Z","updated_at":"2025-07-23T19:06:35.548Z","avatar_url":"https://github.com/michelin.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\".readme/logo-color.svg\" alt=\"KafkaRertryableConsumer\"/\u003e\n\n# Kafka Retryable Consumer\n\n[![GitHub Build](https://img.shields.io/github/actions/workflow/status/michelin/kafka-retryable-consumer/build.yml?branch=main\u0026logo=github\u0026style=for-the-badge)](https://img.shields.io/github/actions/workflow/status/michelin/kafka-retryable-consumer/build.yml)\n[![GitHub release](https://img.shields.io/github/v/release/michelin/kafka-retryable-consumer?logo=github\u0026style=for-the-badge)](https://github.com/michelin/kafka-retryable-consumer/releases)\n[![GitHub Stars](https://img.shields.io/github/stars/michelin/kafka-retryable-consumer?logo=github\u0026style=for-the-badge)](https://github.com/michelin/kafka-retryable-consumer)\n[![SonarCloud Coverage](https://img.shields.io/sonar/coverage/michelin_kafka-retryable-consumer?logo=sonarcloud\u0026server=https%3A%2F%2Fsonarcloud.io\u0026style=for-the-badge)](https://sonarcloud.io/component_measures?id=michelin_kafka-retryable-consumer\u0026metric=coverage\u0026view=list)\n[![SonarCloud Tests](https://img.shields.io/sonar/tests/michelin_kafka-retryable-consumer/main?server=https%3A%2F%2Fsonarcloud.io\u0026style=for-the-badge\u0026logo=sonarcloud)](https://sonarcloud.io/component_measures?metric=tests\u0026view=list\u0026id=michelin_kafka-retryable-consumer)\n[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg?logo=apache\u0026style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)\n\nSimple Kafka consumer library with retry capabilities.\n\nRetryable Consumer Libraries, allow you to easily build a Kafka API consumer with retry capabilities.\nThis lib is proposed to ease the usage of kafka for simple data consumer with external system call.\n\n\u003c/div\u003e\n\n## Table of Contents\n* [Features](#features)\n    * [Current version](#current-version)\n    * [To be implemented](#to-be-implemented)\n* [Getting Started](#getting-started)\n  * [Vanilla Java](#vanilla-java)\n  * [Springboot](#springboot)\n  * [Code References](#code-references)\n\n## Features\n### Current version\n- Error management with Dead Letter Queue\n- Automatic record commit management in case of error or partition rebalance\n- Retry on error with Exception exclusion management\n\n### To be implemented\n- Implement core lib Log\u0026Fail feature\n- Health checks in core library for non springboot projects\n- Springboot starter with core lib health checks integration \n\n\n## Getting started\nKafka consumer libs repository hosts various libraries that might be usefull for your Kafka project.\n- retryable-consumer-core : Core library for easy retryable consumers implementation\n\n### Vanilla Java\n\nInsert retryable-consumer-core dependency in your POM :\n```xml\n\u003cdependency\u003e\n    \u003cgroupId\u003ecom.michelin.kafka\u003c/groupId\u003e\n    \u003cartifactId\u003eretryable-consumer-core\u003c/artifactId\u003e\n    \u003cversion\u003e${retryable-consumer-core.version}\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nThen in your code, on a retryable consumer just call ```java listen(topics)``` method :\n```java\ntry(RetryableConsumer\u003cString, String\u003e retryableConsumer = new RetryableConsumer\u003c\u003e()) {\n    retryableConsumer.listen(\n        Collections.singleton(\"MY_TOPIC\"), //Topic name\n        businessProcessor::processRecord //Function process called by RetryableConsumer for each record\n    );\n}\n```\n`businessProcessor` must be your own processing classe with a method (`processRecord` in this example) accepting one\n`ConsumerRecord\u003cK, V\u003e` as input.\n\n:warning: ```java listen(topics)``` method is blocker and will start an endless loop to consumer you topic record.\nIf you need a non-blocking method, please use ```java listenAsync(topics)``` that returns a future.\n\nIn order to work properly, the retryable consumer will try to load kafka configuration from `application.yml`\nor `application.yaml` or `application.properties`.\n\nExample of application.yml :\n\n```yaml\nkafka:\n  retryable:\n    name: test-retry-consumer # name of the retryable consumer\n    consumer: # All retryable consumer properties\n      not-retryable-exceptions: # List of exceptions that will be ignored by the retry mechanism\n        - java.lang.NullPointerException\n        - java.lang.IllegalArgumentException\n      stop-on-error: false # If true, the consumer will completely stop on not retryable error. Default value = false\n      retry: # Retry configuration\n        max: 10 # Maximum number of retry. 0 means infinite retry. Default value = 0\n      poll:\n        backoff:\n          ms: 2345 # The time, in milliseconds, spent waiting in poll if data is not available in the buffer.\n      properties: # All Kafka server configuration, please add your custom kafka consumer config here\n        application:\n          id: retryable-consumer-test\n        bootstrap:\n          servers: fake.server:9092\n        specific:\n          avro:\n            reader: true\n      topics: TOPIC # List of topics to listen to. Not mandatory\n    dead-letter:\n      producer:\n        properties: # All Kafka server configuration for the dlq producer, please add your custom kafka producer config here\n          application:\n            id: dl-producer-test\n          bootstrap:\n            servers: fake.server:9092\n        topic: DL_TOPIC\n    \n```\n\n## Springboot\n\nA Springboot starter should be build soon. Meanwhile, raw usage of Retryable Consumer CORE lib is possible in springboot.\n\n### Step 1: Plug your Springboot configuration file on \"KafkaRetryableConfiguration\" class\n\nSimply create a configuration that configuration class\n```java\npackage com.michelin.oscp.huw.configuration;\n\nimport com.michelin.kafka.configuration.KafkaRetryableConfiguration;\nimport org.springframework.boot.context.properties.ConfigurationProperties;\nimport org.springframework.context.annotation.Configuration;\n\n@Configuration\n@ConfigurationProperties(prefix = \"kafka.retryable\")\npublic class RetryableConsumerConfiguration extends KafkaRetryableConfiguration {}\n```\n\nIn this example, all retryable consumer configuration should be placed under \"kafka.retryable\" prefix in config file.\n\nExample of application.yml\n```yaml \nkafka:\n  retryable:\n    name: myConsuimer # name of the retryable consumer (not mandatory)\n    consumer: # All retryable consumer properties\n      not-retryable-exceptions:\n        - java.lang.NullPointerException\n        - com.michelin.my.own.project.MyCustomException\n      retry-max: 3 # Retry configuration\n      poll:\n        backoff:\n          ms: 500 # The time, in milliseconds, spent waiting in poll if data is not available in the buffer.\n      properties: # All Kafka server configuration, please add your custom kafka consumer config here\n        auto.offset.reset: earliest\n        bootstrap.servers: mybootstrap-server:9092 # This is kafka dev cluster\n        schema.registry.url: https://my-schema-registry/dev/ # This is kafka dev sr\n        basic.auth.credentials.source: USER_INFO\n        basic.auth.user.info: user:user-secret\n        key.deserializer: org.apache.kafka.common.serialization.StringDeserializer\n        value.deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer\n        specific.avro.reader: true #org.apache.avro.generic.GenericData$Record cannot be cast to HUWOutOrder # to solve this error. I added this config.\n        max.poll.interval.ms: 30000 #5 min\n        max.poll.records: 1\n        enable.auto.commit: false  # is this for both consumer and producer ?\n        fetch.min.bytes: 1\n        heartbeat.interval.ms: 3000\n        isolation.level: read_committed\n        group.id: prefix.myProject\n        sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaUser\" password=\"kafkaSecret\";\n        sasl.mechanism: PLAIN\n        security.protocol: SASL_SSL\n        session.timeout.ms: 45000\n        avro.remove.java.properties: true\n      topics: prefix.myTopic\n    dead-letter:\n      properties: # All Kafka server configuration for the dlq producer, please add your custom kafka producer config\n        bootstrap.servers: mybootstrap-server:9092 # This is kafka dev cluster\n        schema.registry.url: https://my-schema-registry/dev/ # This is kafka dev sr\n        basic.auth.credentials.source: USER_INFO\n        basic.auth.user.info: user:user-secret\n        acks: all\n        avro.remove.java.properties: true\n        enable.idempotence: true\n        sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaUser\" password=\"kafkaSecret\";\n        security.protocol: SASL_SSL\n        key.serializer: org.apache.kafka.common.serialization.StringSerializer\n        value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer\n      topic: prefix.myTopicDeadLetterQ\n```\n\n### Step 2: Build a Kafka consumer service\n\nBuild a springboot service with a method that bootstrap the consumer.\nThen call this method whenever you are ready to start your consumer.\n\nWe advise to put @Async annotation on this method to avoid blocking your springboot app while running\nthe synchronous KafkaRetryableConsumer method ```listen(topics)```. (or use the ```listenAsync(topics)``` method)\n\n```java \n    ...\n    @Autowired\n    private RetryableConsumerConfiguration retryableConsumerConfiguration;\n    ...\n    @Async\n    public void run(){\n        log.info(\"Starting Out Order Event Listener\");\n\n        try(RetryableConsumer\u003cString, MyAvroObject\u003e retryableConsumer = new RetryableConsumer\u003c\u003e(retryableConsumerConfiguration)) {\n            retryableConsumer.listen(\n                    Collections.singleton(\"MY_INPUT_TOPIC\"),\n                    myBusinessProcessService::processRecord\n            );\n        }\n    }\n```\n## Code References\nThis library heavily relies on an original version from @jeanlouisboudart : https://github.com/jeanlouisboudart/retriable-consumer\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichelin%2Fkafka-retryable-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmichelin%2Fkafka-retryable-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmichelin%2Fkafka-retryable-consumer/lists"}