{"id":13783423,"url":"https://github.com/divolte/divolte-kafka-consumer","last_synced_at":"2026-01-10T17:09:55.611Z","repository":{"id":22394750,"uuid":"25731739","full_name":"divolte/divolte-kafka-consumer","owner":"divolte","description":"Helper for consuming Divolte events from Kafka queues and deserializing Avro records into Java objects using Avro's generated code.","archived":true,"fork":false,"pushed_at":"2014-11-06T14:07:51.000Z","size":260,"stargazers_count":15,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-03T19:07:51.032Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/divolte.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-25T13:08:09.000Z","updated_at":"2023-01-28T01:24:13.000Z","dependencies_parsed_at":"2022-08-20T12:20:53.686Z","dependency_job_id":null,"html_url":"https://github.com/divolte/divolte-kafka-consumer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/divolte%2Fdivolte-kafka-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/divolte%2Fdivolte-kafka-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/divolte%2Fdivolte-kafka-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/divolte%2Fdivolte-kafka-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/divolte","download_url":"https://codeload.github.com/divolte/divolte-kafka-consumer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225086489,"owners_count":17418732,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T19:00:21.150Z","updated_at":"2026-01-10T17:09:55.561Z","avatar_url":"https://github.com/divolte.png","language":"Java","funding_links":[],"categories":["Development"],"sub_categories":["Consumers"],"readme":"Divolte Kafka Consumer\n======================\n\nHelper library for writing Kafka consumers that process events created by [Divolte Collector](https://github.com/divolte/divolte-collector). Divolte Collector captures click stream data and translates events into Avro records which are published on Kafka topics. The contents of these messages are the raw bytes produced by serializing the Avro records. This library allows to create consumers for these events in Java in a typesafe manner with minimal boilerplate.\n\nTo use the consumer, you need to generate Java code from your Avro schema. See [divolte-examples/avro-schema](https://github.com/divolte/divolte-examples/tree/master/avro-schema) for an example of a project that uses Maven to build a jar with generated code from a schema.\n\n## Usage\n\nExample consumer code:\n```java\npublic class ConsumerExample {\n    // Event handler that prints records to stdout\n    static class MyEventHandler implements SimpleEventHandler\u003cMyEventRecord\u003e {\n        @Override\n        public void handle(MyEventRecord event) throws Exception {\n            System.out.println(event);\n        }\n    }\n\n    // Supplier of event handler instances\n    static class MyEventHandlerSupplier implements Supplier\u003cSimpleEventHandler\u003cMyEventRecord\u003e\u003e {\n        @Override\n        public SimpleEventHandler\u003cMyEventRecord\u003e get() {\n            return new MyEventHandler();\n        }\n    }\n\n    public static void main(String[] args) {\n        // Create the consumer\n        // MyEventRecord is generated by the Avro Java code generator\n        DivolteKafkaConsumer\u003cMyEventRecord\u003e consumer = DivolteKafkaConsumer.createConsumerWithSimpleHandler(\n                \"divolte\",                             // Kafka topic\n                \"zk1:2181,zk2:2181,zk3:2181\",          // Zookeeper quorum hosts + ports\n                \"my-consumer-group\",                   // Kafka consumer group ID\n                2,                                     // Number of threads for this consumer instance\n                new MyEventHandlerSupplier(),          // Supplier of event handler instances\n                MyEventRecord.getClassSchema());       // Avro schema\n\n        // Add a shutdown hook that stops the consumer\n        // This handles CTRL+C or kill\n        Runtime.getRuntime().addShutdownHook(new Thread(\n                new Runnable() {\n                    @Override\n                    public void run() {\n                        consumer.shutdownConsumer();\n                    }\n                }));\n\n        // Start the consumer\n        consumer.startConsumer();\n    }\n}\n```\n\nIf you are using Java 8, the above example can be condensed to this:\n```java\npublic class ConsumerExample {\n    public static void main(String[] args) {\n        // Create the consumer\n        DivolteKafkaConsumer\u003cMyEventRecord\u003e consumer = DivolteKafkaConsumer.createConsumerWithSimpleHandler(\n                \"divolte\",                           // Kafka topic\n                \"zk1:2181,zk2:2181,zk3:2181\",        // Zookeeper quorum hosts + ports\n                \"my-consumer-group\",                 // Kafka consumer group ID\n                2,                                   // Number of threads for this consumer instance\n                () -\u003e (e) -\u003e System.out.println(e),  // Supplier of event handler instances\n                MyEventRecord.getClassSchema());     // Avro schema\n\n        // Add a shutdown hook that stops the consumer\n        // This handles CTRL+C or kill\n        Runtime\n        .getRuntime()\n        .addShutdownHook(\n                new Thread(consumer::shutdownConsumer)\n                );\n\n        // Start the consumer\n        consumer.startConsumer();\n    }\n}\n```\n\nFor a more complete usage example, have a look at [divolte-examples/tcp-kafka-consumer](https://github.com/divolte/divolte-examples/tree/master/tcp-kafka-consumer).\n\n## Build from source\nWe use [Gradle](http://www.gradle.org/) as a build tool. You need Java 7 or higher to build.\n\nTo build from source on your machine:\n\n```sh\n# cd into your preferred working dir\ngit clone https://github.com/divolte/divolte-kafka-consumer.git\ncd divolte-kafka-consumer\n\n# build the source\n./gradlew build\n\n# generate Eclipse project files\n./gradlew eclipse\n\n# install into your local Maven repository\n./gradlew install\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdivolte%2Fdivolte-kafka-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdivolte%2Fdivolte-kafka-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdivolte%2Fdivolte-kafka-consumer/lists"}