https://github.com/rishant/kafka-java-example
sample producers and consumers
https://github.com/rishant/kafka-java-example
Last synced: 10 months ago
JSON representation
sample producers and consumers
- Host: GitHub
- URL: https://github.com/rishant/kafka-java-example
- Owner: rishant
- Created: 2024-11-24T19:15:09.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-24T19:59:11.000Z (about 1 year ago)
- Last Synced: 2024-11-24T20:27:35.729Z (about 1 year ago)
- Language: Java
- Size: 19.5 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Kafka Consumer Offset Management: `commitSync` vs `commitAsync`
This document provides guidance on using the `commitSync` and `commitAsync` methods in Kafka consumers for managing offsets efficiently based on application requirements.
## **Overview**
Kafka consumers use offsets to track the progress of message consumption from partitions. These offsets can be committed to the broker using either:
- **`commitSync`**: Synchronously commits offsets.
- **`commitAsync`**: Asynchronously commits offsets.
Choosing between the two depends on your application's trade-offs between reliability and performance.
---
## **When to Use `commitSync`**
The `commitSync` method blocks the consumer until the broker confirms the offsets are successfully committed. It ensures stronger consistency.
### **Use Cases**
1. **Critical Applications**:
- Financial transactions, sensitive data processing.
- Losing or reprocessing messages is unacceptable.
2. **Batch Processing**:
- After processing a batch, commit the offsets to avoid reprocessing.
3. **Error Recovery**:
- For fallback mechanisms when `commitAsync` fails.
4. **Low Message Throughput**:
- Suitable when a slight latency is acceptable due to lower message volumes.
### **Advantages**
- Guarantees offsets are committed successfully.
- Simplifies error handling.
### **Disadvantages**
- Slower due to blocking nature, impacting performance.
---
## **When to Use `commitAsync`**
The `commitAsync` method commits offsets without blocking, allowing the consumer to continue processing messages immediately.
### **Use Cases**
1. **High Throughput Applications**:
- Systems prioritizing performance, such as analytics or logging.
2. **Non-Critical Applications**:
- Occasional duplicate processing is acceptable.
3. **Performance-Critical Scenarios**:
- Reduces latency caused by blocking calls.
4. **Frequent Offset Commit**:
- Suitable when offsets need frequent updates.
### **Advantages**
- High performance with non-blocking behavior.
- Ideal for low-latency requirements.
### **Disadvantages**
- No guarantee of successful offset commits.
- Requires additional error handling.
---
## **Combining `commitSync` and `commitAsync`**
For most applications, you can combine both methods to balance performance and reliability:
1. **Use `commitAsync`** during normal operation to optimize throughput.
2. **Use `commitSync`** during shutdown or critical moments to ensure offsets are safely committed.
---
## **Sample Code**
```java
try {
while (true) {
ConsumerRecords records = consumer.poll(Duration.ofMillis(1000));
// Process messages
for (ConsumerRecord record : records) {
System.out.printf("Consumed message: key=%s, value=%s, offset=%d%n",
record.key(), record.value(), record.offset());
}
// Commit offsets asynchronously for performance
consumer.commitAsync((offsets, exception) -> {
if (exception == null) {
System.out.println("Offsets committed asynchronously: " + offsets);
} else {
System.err.println("Commit failed: " + exception.getMessage());
}
});
}
} catch (Exception e) {
System.err.println("Error during consumption: " + e.getMessage());
} finally {
try {
// Ensure final offsets are committed synchronously before shutdown
consumer.commitSync();
System.out.println("Offsets committed synchronously during shutdown.");
} catch (Exception ex) {
System.err.println("Error during final commit: " + ex.getMessage());
} finally {
consumer.close();
}
}
```
---
## **Decision Guide**
| **Criteria** | **Use `commitSync`** | **Use `commitAsync`** |
|------------------------------|-----------------------------------------|---------------------------------------|
| **Consistency Requirements** | Critical for consistency | Consistency is less critical |
| **Throughput** | Lower throughput acceptable | High throughput required |
| **Message Importance** | Loss of messages is unacceptable | Occasional duplicate processing okay |
| **Application Type** | Financial transactions, critical data | Analytics, logging, non-critical data |
---
## **Best Practices**
1. **Logging in `commitAsync`**:
- Always provide a callback to handle commit failures for visibility.
2. **Graceful Shutdown**:
- Use `commitSync()` before shutting down the consumer to avoid data loss.
3. **Retry Logic**:
- Combine `commitAsync` with fallback logic using `commitSync` in case of repeated failures.
By understanding the trade-offs and combining these methods as needed, you can ensure your Kafka consumer application is both efficient and reliable.