{"id":19614874,"url":"https://github.com/leancloud/kafka-java-consumer","last_synced_at":"2026-06-11T03:31:17.153Z","repository":{"id":45150238,"uuid":"230073616","full_name":"leancloud/kafka-java-consumer","owner":"leancloud","description":"A Kafka consumer which can help you to overcome common pitfalls","archived":false,"fork":false,"pushed_at":"2022-01-05T01:10:23.000Z","size":227,"stargazers_count":5,"open_issues_count":1,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-11-21T03:35:20.038Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leancloud.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-25T08:55:11.000Z","updated_at":"2025-01-12T05:19:40.000Z","dependencies_parsed_at":"2022-08-25T16:12:44.436Z","dependency_job_id":null,"html_url":"https://github.com/leancloud/kafka-java-consumer","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/leancloud/kafka-java-consumer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leancloud%2Fkafka-java-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leancloud%2Fkafka-java-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leancloud%2Fkafka-java-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leancloud%2Fkafka-java-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leancloud","download_url":"https://codeload.github.com/leancloud/kafka-java-consumer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leancloud%2Fkafka-java-consumer/sbom","scorecard":{"id":581808,"data":{"date":"2025-08-11","repo":{"name":"github.com/leancloud/kafka-java-consumer","commit":"f22719995c57eb66a411b98b1ed71e2dbad940f9"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.9,"checks":[{"name":"Code-Review","score":0,"reason":"Found 0/11 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.md:0","Info: FSF or OSI recognized license: MIT License: LICENSE.md:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":6,"reason":"4 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-55g7-9cwv-5qfv","Warn: Project is vulnerable to: GHSA-fjpj-2g6w-x25r","Warn: Project is vulnerable to: GHSA-pqr6-cmr2-h8hf","Warn: Project is vulnerable to: GHSA-qcwq-55hx-v3vh"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 24 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-20T19:22:16.907Z","repository_id":45150238,"created_at":"2025-08-20T19:22:16.907Z","updated_at":"2025-08-20T19:22:16.907Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34181554,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T10:54:21.112Z","updated_at":"2026-06-11T03:31:17.134Z","avatar_url":"https://github.com/leancloud.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kafka Java Consumer\n\n[![Build Status](https://api.travis-ci.org/leancloud/kafka-java-consumer.svg?branch=master)](https://travis-ci.org/leancloud/kafka-java-consumer)\n[![Coverage Status](https://codecov.io/gh/leancloud/kafka-java-consumer/branch/master/graph/badge.svg)](https://codecov.io/gh/leancloud/kafka-java-consumer)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Maven](https://img.shields.io/github/release/leancloud/kafka-java-consumer.svg)](https://github.com/leancloud/kafka-java-consumer/releases)\n\nKafka provides a Java Kafka Client to communicate with it. It's a greate lib which is very versatile and flexible, but many things may go wrong if you use it without good care or good understanding about Kafka internals. We will talk about some of the common pitfalls on the consumer side which are easily to encounter with and this lib is used to help you to overcome them peacefully.\n\nUsually, after we have subscribed the consumer to some topics, we need a loop to do these things: \n\n* Fetch records from Kafka broker by using `poll` method on `KafkaConsumer`;\n* Process the fetched records;\n* Commit the offset of these fetched records, so they will not be consumed again;\n\nWe need to call `poll` constantly and ensure that the interval between each call should not too long, otherwise after a session timeout or a poll timeout, the broker may think our consumer is not alive and revoke every partitions assigned to our consumer. If we need to do a lot of things with the records we fetched, we may need to set the Kafka consumer configuration `max.poll.interval.ms` to a comparatively larger value to give us enough time to process all these records. But it's not trival to set `max.poll.interval.ms` to a large value. The larger the `max.poll.interval.ms` value is, the longer time it's needed for a broker to realize that a consumer is dead when something wrong with the consuemr. In addition to tune the `max.poll.interval.ms` configuration, we can spare the polling thread only to poll records from broker and submit all the fetched records to a thread pool which is taking charge of processing these records. But to do it in this way, we need to pause the partitions of all the fetched records before processing them to prevent the polling threads from polling more records while the previous records are still processing. Of course, we should remember to resume a paused parition after we have processed all records from that partition. Futher more, after a partition reassignment, we should remember which partition we paused before the partition reassignemnt, and pause the paused partition again. \n\nKafka Client provides a synchronous and a asynchronous way to commit offset of records. In addition to them, Kafka Client also provides a way to commit for specific partition and offset, and a way to commit all the records fetched at once. We should remember to commit all the processed records from a partition before this partition is revoked. We should remember to commit all the processed records before the consuemr shutdown. If we commit offset for a specific record, we should remember to plus one to the offset of that record, such as assuming the record to commit have partition 0 and offset 100, we should commit partition 0 to 101 instead of 100, otherwise that processed records will be fetched again. If a consumer were assigned a parition which have no records for a long time, we should still remember to commit the committed offset of that partition periodically, otherwise after the commit log of that partition was removed from broker, because of retention timeout, broker will not remember where the commit offset of that partition for the consumer was. If the consumer set Kafka configuration `auto.offset.reset` to **earliest**, after a reboot, the cosumer will poll all the records from the partition for which broker forgot where we committed and process all of them over again.\n\nAll in all, Kafka Client is not a tool which can be used directly without good care and doing some research. But with the help of this lib, you can consume records from a subscribed topic and process them with or without a deidicated thread pool more safely and easily. It encapsulates loads of best practices to acheive that goal.\n\n## Usage\n\nFirstly, we need configurations for Kafka consumer. For example:\n\n```Java\nfinal Map\u003cString, Object\u003e configs = new HashMap\u003c\u003e();\nconfigs.put(\"bootstrap.servers\", \"localhost:9092\");\nconfigs.put(\"group.id\", \"LeanCloud\");\nconfigs.put(\"auto.offset.reset\", \"earliest\");\nconfigs.put(\"max.poll.records\", 10);\nconfigs.put(\"max.poll.interval.ms\", 30_000);\nconfigs.put(\"key.deserializer\", \"...\");\nconfigs.put(\"value.deserializer\", \"...\");\n```\n\nThen, define how you need to handle a record consumed from Kafka. Here we just log the consumed record:\n\n```java\nConsumerRecordHandler\u003cInteger, String\u003e handler = record -\u003e {\n    logger.info(\"I got a record: {}\", record);\n};\n```\n\nNext, we need to choose the type of consumer to use. We have five kinds of consumers and each of them have different committing policy. Here is a simple specification for them:\n\ncommit policy | description\n------ | ------------\nautomatic commit | Commit offsets of records fetched from broker automatically in a fixed interval. \nsync commit | Commit offsets synchronously only when all the fetched records have been processed. \nasync commit | Commit offsets asynchronously only when all the fetched records have been processed. If there are too many pending async commit requests or the last async commit request was failed, it'll switch to synchronous mode to commit synchronously and switch back when the next synchoronous commit success.\npartial sync commit | Whenever there is a processed consumer record, only those records that have already been processed are committed synchronously, leaving the ones that have not been processed yet to be committed. \npartial async commit | Whenever there is a processed consumer record, only those records that have already been processed are committed asynchronously, leaving the ones that have not been processed yet to be committed. If there are too many pending async commit requests or the last async commit request was failed, it'll switch to synchronous mode to commit synchronously and switch back when the next synchoronous commit success.\n\nTaking sync-committing consumer as an example, you can create a consumer with a thread pool and subscribe it to a topic like this:\n\n```java\nfinal LcKafkaConsumer\u003cInteger, String\u003e consumer = LcKafkaConsumerBuilder\n                .newBuilder(configs, handler)\n                // true means the LcKafkaConsumer should shutdown the input thread pool when it is shutting down\n                .workerPool(Executors.newCachedThreadPool(), true)  \n                .buildSync();\nconsumer.subscribe(Collections.singletonList(\"LeanCloud-Topic\"));\n```\n\nPlease note that we passed a `ExecutorService` to build the `LcKafkaConsumer`, all the records consumed from the subscribed topic will be handled by this `ExecutorService` using the input `ConsumerRecordHandler`. \n\nWhen we are done with this consumer, we need to close it:\n\n```\nconsumer.close()\n```\n\nFor all the APIs and descriptions of all the kinds of consumers, please refer to the Java Doc.\n\n## License\n\nCopyright 2020 LeanCloud. Released under the [MIT License](https://github.com/leancloud/filter-service/blob/master/LICENSE.md).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleancloud%2Fkafka-java-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleancloud%2Fkafka-java-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleancloud%2Fkafka-java-consumer/lists"}