{"id":19028724,"url":"https://github.com/researchgate/kafka-metamorph","last_synced_at":"2026-07-22T06:35:08.376Z","repository":{"id":141869992,"uuid":"58392983","full_name":"researchgate/kafka-metamorph","owner":"researchgate","description":"Apache Kafka consumer API for selective partition consumption and explicit offset control","archived":false,"fork":false,"pushed_at":"2016-11-17T17:29:25.000Z","size":147,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-06-23T15:50:23.777Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/researchgate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-05-09T16:51:37.000Z","updated_at":"2016-12-19T14:29:36.000Z","dependencies_parsed_at":null,"dependency_job_id":"2ccab659-8b6b-42b2-8101-4e900a0e5721","html_url":"https://github.com/researchgate/kafka-metamorph","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/researchgate/kafka-metamorph","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fkafka-metamorph","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fkafka-metamorph/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fkafka-metamorph/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fkafka-metamorph/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/researchgate","download_url":"https://codeload.github.com/researchgate/kafka-metamorph/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/researchgate%2Fkafka-metamorph/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263493374,"owners_count":23475184,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T21:12:09.057Z","updated_at":"2025-10-11T23:11:48.727Z","avatar_url":"https://github.com/researchgate.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-metamorph\nBackwards compatible Apache Kafka consumer API for selective partition consumption and explicit offset control\n\n[![Build Status](https://travis-ci.org/researchgate/kafka-metamorph.svg?branch=master)](https://travis-ci.org/researchgate/kafka-metamorph)\n[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0)\n\n- [Background](#background)\n- [Usage](#usage)\n\n## Background\n\n### TL;DR\n\nWith the introduction of Kafka 0.9.0 there are now 3 incompatible consumer interfaces and this project tries to unify them (and provide a consistent API which can be used with several Kafka versions).\n\nSee also https://xkcd.com/927/\n\n### The old fashioned way\n\nBefore the introduction of version 0.9.0 Apache Kafka provided two consumer interfaces,\nthe [SimpleConsumer API](http://kafka.apache.org/082/documentation.html#simpleconsumerapi) and the [high-level Consumer API](http://kafka.apache.org/082/documentation.html#highlevelconsumerapi).\n\nWhile the high-level consumer provided a convenient interface and also took care of offset management, it had a major drawback:\nit did not allow explicit offset control (e.g. no possibility to re-process messages) and also it was not possible to consume\nonly from a subset from partitions.\n\nThe SimpleConsumer provided this functionality but in order to use this consumer properly it was neccessary to create a\nlot of boilerplate code for error handling and for dealing with broker leader changes (see [official Kafka 0.8.0 SimpleConsumer example](https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example)).\n\nUnfortunately both consumer APIs were not compatible and it was rather tedious operation to replace one implementation with the other.\n\n### Brave new world - Kafka 0.9.0\n\nWith the release of Kafka 0.9.0 a [new unified consumer API](http://kafka.apache.org/090/documentation.html#consumerapi) was introduced which addressed the issues described above.\nThis new interface combined the automatic offset management while still providing means to bind only to a selection of partitions\nand allowed external offset control.\n\nSo why don't we just switch to the Kafka 0.9.0 consumer and stop worrying?\n\nUnfortunately with the release of Kafka 0.9.0 there was also a change within the inter-broker protocol and therefore\nthe new unified consumer interface cannot be used with older Kafka broker versions.\n\nIf you have an infrastructure which is running Kafka brokers with a version lower than 0.9.0 then you are between\na rock and a hard place.\n\nEither you start using one of the deprecated (but still supported) consumer interfaces and you\nneed to refactor your codebase later or you need to wait until your brokers can be updated to 0.9.x\n\n### Introducing metamorph\n\nThis is the part in which kafka-metamorph comes into play. By providing one unified interface which can be\nused with older Kafka versions you can implement your use-case today and worry about broker updates later. Yay!\n\n## Usage\n\nThe project is separated into a generic interface (provided via metamorph-common) and implementations targeted at a specific Kafka version (e.g. metamorph-kafka-08 for usage with Kafka 0.8.x).\n\nThe [PartitionConsumer](http://researchgate.github.io/kafka-metamorph/latest/javadoc/metamorph-common/index.html?net/researchgate/kafka/metamorph/PartitionConsumer.html) interface provided by metamorph-common is an abstraction layer on top of different Kafka consumer implementations. It allows the discovery of all partitions of a topic and the consumption of a given topic partition.\n\nThis example shows how to fetch data from partition 0 of the topic \"some_topic\".\n\n```\n    PartitionConsumer\u003cString, String\u003e consumer = new Kafka08PartitionConsumer\u003c\u003e(new Kafka08PartitionConsumerConfig.Builder(\"localhost:9092\").build(), new StringDecoder(null), new StringDecoder(null));\n\n    consumer.assign(new TopicPartition(\"some_topic\", 0));\n\n    for (PartitionConsumerRecord\u003cString,String\u003e record : consumer.poll(0)) {\n        System.out.println(record.offset() + \",\" + record.key() + \",\" + record.value());\n    }\n```\n\nThe initialization does contain version specific code (in this example for Kafka 0.8.x) but everything after the initialization is version agnostic and will work with newer and older Kafka versions.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresearchgate%2Fkafka-metamorph","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresearchgate%2Fkafka-metamorph","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresearchgate%2Fkafka-metamorph/lists"}