https://github.com/datanerds-io/verteiler

Meine dispatcher says there's something wrong mit deine Kabel. Java 8 BlockingQueue Kafka Consumer with automatically maintained offset.
https://github.com/datanerds-io/verteiler

consumer java kafka kafka-consumer

Last synced: 6 months ago
JSON representation

Meine dispatcher says there's something wrong mit deine Kabel. Java 8 BlockingQueue Kafka Consumer with automatically maintained offset.

• Host: GitHub
• URL: https://github.com/datanerds-io/verteiler
• Owner: datanerds-io
• License: apache-2.0
• Created: 2016-10-28T00:27:52.000Z (over 9 years ago)
• Default Branch: develop
• Last Pushed: 2018-08-04T05:01:43.000Z (almost 8 years ago)
• Last Synced: 2023-07-03T22:36:05.504Z (almost 3 years ago)
• Topics: consumer, java, kafka, kafka-consumer
• Language: Java
• Homepage:
• Size: 32.2 KB
• Stars: 7
• Watchers: 10
• Forks: 3
• Open Issues: 1
• Metadata Files:
  • Readme: README.MD
  • License: LICENSE

Awesome Lists containing this project

README

          
[![Build Status](https://api.travis-ci.org/datanerds-io/verteiler.svg?branch=develop)](https://travis-ci.org/datanerds-io/verteiler)

[![Maven Central](https://img.shields.io/maven-central/v/io.datanerds/verteiler.svg)](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22io.datanerds%22%20a%3A%22verteiler%22)

# Verteiler (fɛɐˈtailɐ)

"Verteiler" is german and translates to _distributor_. When you are using the (new) `kafka-clients` consumer introduced with v0.9.0, you will face the issue that `Consumer.poll(...)` can only be called from [one thread][1]. Verteiler leverages the kafka-client consumer implementation to distribute messages from assigned partitions to `BlockingQueue`s. Each assigned partition will relay its messages to its own internal queue. These queues have worker threads which process the message leveraging a user defined consumer. Once a message has been processed successfully its offset will be committed asynchronously after `poll(...)` has finished. This way only the offset of processed messages is committed.

# How to use `verteiler`

## `BlockingQueueConsumer`

### Parameters

`String topic`: Name of the topic you want to consume messages from.

`Properties kafkaConfig`: Regular [consumer config][2]. By default `verteiler` will disable auto offset commit and set a client id when not given.

`int queueSize`: Size of the internal queue, which depends on your needs.

`java.util.function.Consumer action`: The function which will handle a message.

### Example

Simple message counter:

```java

Properties props = new Properties();

props.setProperty(BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");

...

AtomicInteger messageCounter = new AtomicInteger();

Consumer action = (message) -> messageCounter.incrementAndGet();

BlockingQueueConsumer consumer = new BlockingQueueConsumer<>("my_topic", props, 42, action);

consumer.start();

```

For a full example take a look at [BlockingQueueConsumerTest][3].

# Build

`gradle build` will build the project. The integration tests will start embedded zookeeper & kafka instances.

[1]: https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#multithreaded

[2]: https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/ConsumerConfig.html

[3]: https://github.com/datanerds-io/verteiler/blob/develop/it-test/src/test/java/io/datanerds/verteiler/it_test/BlockingQueueConsumerTest.java