{"id":27047198,"url":"https://github.com/rishant/kafka-java-example","last_synced_at":"2025-04-05T06:28:40.079Z","repository":{"id":264515046,"uuid":"893567857","full_name":"rishant/kafka-java-example","owner":"rishant","description":"sample producers and consumers","archived":false,"fork":false,"pushed_at":"2024-11-24T19:59:11.000Z","size":20,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-11-24T20:27:35.729Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rishant.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-24T19:15:09.000Z","updated_at":"2024-11-24T19:59:14.000Z","dependencies_parsed_at":"2024-11-24T20:28:16.165Z","dependency_job_id":"63d73fe4-a5c0-44ee-9ce1-702ab5fb5154","html_url":"https://github.com/rishant/kafka-java-example","commit_stats":null,"previous_names":["rishant/kafka-java-example"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rishant%2Fkafka-java-example","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rishant%2Fkafka-java-example/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rishant%2Fkafka-java-example/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rishant%2Fkafka-java-example/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rishant","download_url":"https://codeload.github.com/rishant/kafka-java-example/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299353,"owners_count":20916171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-05T06:28:39.120Z","updated_at":"2025-04-05T06:28:40.070Z","avatar_url":"https://github.com/rishant.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# Kafka Consumer Offset Management: `commitSync` vs `commitAsync`\n\nThis document provides guidance on using the `commitSync` and `commitAsync` methods in Kafka consumers for managing offsets efficiently based on application requirements.\n\n## **Overview**\n\nKafka consumers use offsets to track the progress of message consumption from partitions. These offsets can be committed to the broker using either:\n- **`commitSync`**: Synchronously commits offsets.\n- **`commitAsync`**: Asynchronously commits offsets.\n\nChoosing between the two depends on your application's trade-offs between reliability and performance.\n\n---\n\n## **When to Use `commitSync`**\n\nThe `commitSync` method blocks the consumer until the broker confirms the offsets are successfully committed. It ensures stronger consistency.\n\n### **Use Cases**\n1. **Critical Applications**:\n   - Financial transactions, sensitive data processing.\n   - Losing or reprocessing messages is unacceptable.\n2. **Batch Processing**:\n   - After processing a batch, commit the offsets to avoid reprocessing.\n3. **Error Recovery**:\n   - For fallback mechanisms when `commitAsync` fails.\n4. **Low Message Throughput**:\n   - Suitable when a slight latency is acceptable due to lower message volumes.\n\n### **Advantages**\n- Guarantees offsets are committed successfully.\n- Simplifies error handling.\n\n### **Disadvantages**\n- Slower due to blocking nature, impacting performance.\n\n---\n\n## **When to Use `commitAsync`**\n\nThe `commitAsync` method commits offsets without blocking, allowing the consumer to continue processing messages immediately.\n\n### **Use Cases**\n1. **High Throughput Applications**:\n   - Systems prioritizing performance, such as analytics or logging.\n2. **Non-Critical Applications**:\n   - Occasional duplicate processing is acceptable.\n3. **Performance-Critical Scenarios**:\n   - Reduces latency caused by blocking calls.\n4. **Frequent Offset Commit**:\n   - Suitable when offsets need frequent updates.\n\n### **Advantages**\n- High performance with non-blocking behavior.\n- Ideal for low-latency requirements.\n\n### **Disadvantages**\n- No guarantee of successful offset commits.\n- Requires additional error handling.\n\n---\n\n## **Combining `commitSync` and `commitAsync`**\n\nFor most applications, you can combine both methods to balance performance and reliability:\n1. **Use `commitAsync`** during normal operation to optimize throughput.\n2. **Use `commitSync`** during shutdown or critical moments to ensure offsets are safely committed.\n\n---\n\n## **Sample Code**\n\n```java\ntry {\n    while (true) {\n        ConsumerRecords\u003cString, String\u003e records = consumer.poll(Duration.ofMillis(1000));\n        \n        // Process messages\n        for (ConsumerRecord\u003cString, String\u003e record : records) {\n            System.out.printf(\"Consumed message: key=%s, value=%s, offset=%d%n\", \n                record.key(), record.value(), record.offset());\n        }\n\n        // Commit offsets asynchronously for performance\n        consumer.commitAsync((offsets, exception) -\u003e {\n            if (exception == null) {\n                System.out.println(\"Offsets committed asynchronously: \" + offsets);\n            } else {\n                System.err.println(\"Commit failed: \" + exception.getMessage());\n            }\n        });\n    }\n} catch (Exception e) {\n    System.err.println(\"Error during consumption: \" + e.getMessage());\n} finally {\n    try {\n        // Ensure final offsets are committed synchronously before shutdown\n        consumer.commitSync();\n        System.out.println(\"Offsets committed synchronously during shutdown.\");\n    } catch (Exception ex) {\n        System.err.println(\"Error during final commit: \" + ex.getMessage());\n    } finally {\n        consumer.close();\n    }\n}\n```\n\n---\n\n## **Decision Guide**\n\n| **Criteria**                 | **Use `commitSync`**                    | **Use `commitAsync`**                 |\n|------------------------------|-----------------------------------------|---------------------------------------|\n| **Consistency Requirements** | Critical for consistency               | Consistency is less critical          |\n| **Throughput**               | Lower throughput acceptable            | High throughput required              |\n| **Message Importance**       | Loss of messages is unacceptable       | Occasional duplicate processing okay  |\n| **Application Type**         | Financial transactions, critical data  | Analytics, logging, non-critical data |\n\n---\n\n## **Best Practices**\n1. **Logging in `commitAsync`**:\n   - Always provide a callback to handle commit failures for visibility.\n2. **Graceful Shutdown**:\n   - Use `commitSync()` before shutting down the consumer to avoid data loss.\n3. **Retry Logic**:\n   - Combine `commitAsync` with fallback logic using `commitSync` in case of repeated failures.\n\nBy understanding the trade-offs and combining these methods as needed, you can ensure your Kafka consumer application is both efficient and reliable.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishant%2Fkafka-java-example","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frishant%2Fkafka-java-example","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishant%2Fkafka-java-example/lists"}