{"id":22059235,"url":"https://github.com/opentable/otj-kafka","last_synced_at":"2025-07-24T00:31:42.648Z","repository":{"id":42058833,"uuid":"55091713","full_name":"opentable/otj-kafka","owner":"opentable","description":"Kafka integration library","archived":true,"fork":false,"pushed_at":"2023-09-19T21:45:55.000Z","size":581,"stargazers_count":4,"open_issues_count":0,"forks_count":2,"subscribers_count":26,"default_branch":"master","last_synced_at":"2025-07-12T04:08:10.970Z","etag":null,"topics":["headers","kafka","logging","metrics","platform-java","spring","used-by-eda"],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/opentable.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-03-30T19:34:17.000Z","updated_at":"2024-11-24T17:45:58.000Z","dependencies_parsed_at":"2024-11-30T17:38:53.144Z","dependency_job_id":null,"html_url":"https://github.com/opentable/otj-kafka","commit_stats":null,"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"purl":"pkg:github/opentable/otj-kafka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentable%2Fotj-kafka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentable%2Fotj-kafka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentable%2Fotj-kafka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentable%2Fotj-kafka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/opentable","download_url":"https://codeload.github.com/opentable/otj-kafka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/opentable%2Fotj-kafka/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266774717,"owners_count":23982246,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-23T02:00:09.312Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["headers","kafka","logging","metrics","platform-java","spring","used-by-eda"],"created_at":"2024-11-30T17:27:41.989Z","updated_at":"2025-07-24T00:31:42.176Z","avatar_url":"https://github.com/opentable.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"Kafka integrations and convenience libraries.\n\nKafka Builders\n====\n\nWe provide builders for KafkaConsumer and KafkaProducer.\nWhy would you use them?\n\n* Spring friendly\n* Fluent API, to make most Kafka properties a little more readable and strongly typed\n* Integration with metrics - automatically forwards Kafka metrics to graphite\n* OTL logging - on a sampled basis.\n* Support for injecting raw properties via Spring configuration files.\n* Future support for embedded authentication methods to brokers.\n\nUsage:\n\nAdd `@InjectKafkaBuilderBean` to any Configuration class. You'll now have an extra bean in your context:\n\n* KafkaBuilderFactoryBean\n\nYou may inject them into any class, and use them. The general idiom they follow looks something like this\n\n```\n final Consumer\u003cbyte[], ABEvent\u003e consumer = kafkaBuilderFactoryBean\n                .consumerBuilder(\"nameSuppliedByUser\") // name must be unique per machine. This works because of the deterministic setup. Use producerBuilder for a producer.\n                .withBootstrapServers(Arrays.asList(brokerList().split(\",\")))\n                .withAutoCommit(false)\n                .withClientId(makeClientId(topicPartition))\n                .withGroupId(getKafkaGroupId())\n                .withAutoOffsetReset(autoOffsetResetMode.getType())\n                .withDeserializers(keyDeserializer, abEventDeserializer)\n                .build();\n                ;\n\n```\n\nThis will build the Kafka producer/consumer using the following logic:\n\n* If there's any `ot.kafka.consumer|producer.nameSuppliedByUser` namespaced configuration properties, use these. They\nwill take precedence over the fluent api\n* We preassign client-id in the bean as `name.(serviceInfo.name).(incrementingNumber)`. You may override it as shown above, if desired.\n* Add in any thing specified in the fluent api\n* Wire in metrics and logging. To disable these use `disableLogging()` and/or `disableMetrics()`. The logging only\nkicks in once per 10 seconds, but you may change this rate via `withLoggingSampleRate()` (for example setting to 10 will make it rate limit\nonce per second, changing to 100 will have a rate limit of 10 per second, etc) - mind our logging cluster though!\n* Return the Kafka consumer/producer (we return the interface type Consumer/Producer instead of the implementations KafkaConsumer/KafkaProducer)\n\nThe BuilderFactoryBean is thread safe, but the underlying Builder is NOT.\n\n**About Metrics**\n\n* We preset the root name space as `kafka.consumer|producer.(nameSuppliedByUser)`.\n* If you create multiple consumers and producers in an application, you need to supply stable, reasonable names, if\nyou want to want to have stable metrics to dashboard and alert upon. A naive strategy of a simple auto incrementing\nnumber will only work if you build the Producer/consumers in a deterministic order.\n* You can override the namespace entirely by calling `withMetricRegistry(metricRegistry, prefix)`\n\n**About KafkaStreams**\n\nWe do not yet support KafkaStreams, however a WIP implementations is in the\n`otj-kafka-streams` module.\n\n**About testing***\n\nSee `otj-kafka-tests` discussion below.\n\n\notj-kafka-tests\n====\nThis module includes embedded versions of Kafka and Zookeeper, making tests simple and self contained.\n\nHere is a typical usage:\n\n\n```$xslt\npublic class KafkaBrokerRuleTest {\n    private static final String TEST_TOPIC = \"test-topic\";\n    private static final String TEST_VALUE = \"The quick brown fox jumps over the lazy dog.\";\n\n    @Rule\n    public final EmbeddedKafkaRule kb = new EmbeddedKafkaBuilder()\n            .withTopics(TEST_TOPIC)\n            .rule();\n\n    @Test(timeout = 30000)\n    public void testKafkaRule() throws Exception {\n        EmbeddedKafkaBroker ekb = kb.getBroker();\n\n        try (KafkaProducer\u003cString, String\u003e producer = ekb.createProducer()) {\n            producer.send(new ProducerRecord\u003c\u003e(TEST_TOPIC, TEST_VALUE));\n        }\n\n        try (KafkaConsumer\u003cString, String\u003e consumer = ekb.createConsumer(\"test\")) {\n            consumer.subscribe(Collections.singletonList(TEST_TOPIC));\n            ConsumerRecords\u003cString, String\u003e records = consumer.poll(Duration.ofSeconds(5));\n            assertEquals(1, records.count());\n            assertEquals(TEST_VALUE, records.iterator().next().value());\n        }\n    }\n}\n```\n\nNote that the rule can also be used to get the connection strings directly to be used in the Kafka Builders\nwe have provided\n\n```$xslt\n    final EmbeddedKafkaBroker ekb = kb.getBroker();\n    final String connectionString = ekb.getKafkaBrokerConnect();\n    final Consumer\u003cbyte[], ABEvent\u003e consumer = kafkaConsumerBuilderFactoryBean.\u003cbyte[],ABEvent\u003e\n                    builder(\"nameSuppliedByUser\") // name must be unique per machine. This works because of the deterministic setup\n                    .withBootstrapServers(Arrays.asList(connectionString))\n                    .withAutoCommit(false)\n                    .withClientId(makeClientId(topicPartition))\n                    .withGroupId(getKafkaGroupId())\n                    .withAutoOffsetReset(autoOffsetResetMode.getType())\n                    .withDeserializers(keyDeserializer, abEventDeserializer)\n                    .build();\n                    ;\n```\n\notj-kafka-streams\n===\n\nLogProgressRestoreListener\n--------\n\nA StateRestoreListener for logging the progress of State rebuilds with KafkaStreams\n\n\nOther Tools\n====\n\nOffset Metrics\n--------------\n\nWe provide a class `OffsetMetrics` that adds metrics to a\n`MetricRegistry` instrumenting the size (maximum offset) of given topics\nalong with the offset and lag (size minus offset) of consumer groups'\nconsumption of those topics.\n\nUse is simple.  If using Spring, simply add a new `@Bean`.  Here is an\nexample from chat.\n\n    @Bean\n    OffsetMetrics offsetMetrics(\n            final MetricRegistry metricRegistry,\n            final KafkaConnectionConfiguration connConfig) {\n        final String metricPrefix = \"chat.offset-metrics\";\n        final String groupId = ChatConstants.APP_ID;\n        return OffsetMetrics\n                .builder(\n                    metricPrefix, metricRegistry,\n                    groupId, connConfig.getConnectString()\n                )\n                .addTopics(\n                    connConfig.getRequestTopic(),\n                    connConfig.getMessageTopic()\n                )\n                .build();\n    }\n\n`OffsetMetrics` registers its metrics on startup, so it requires a reference\nto a `MetricRegistry` and a prefix for the metrics it'll register. Metrics\nwill be registered under the namespace `kafka.\u003cprefix\u003e.\u003ctopic\u003e\". The prefix\ncannot contain a period.\n\nIf not using Spring, just be sure to call `start` after construction, and\n`stop` before disposal or shutdown.\n\nThe set of instrumented metrics from one instance is complete.  That is,\nyou don't need to run more than one of these.  Consequently, you may\nwant to run it in a manager / sidecar sort of service.  Or you may want\nto selectively run it from only one of your service instances.  Or you\nmight just not worry about it, and read off the metrics from one\ninstance only.\n\nFor more detail, see the extensive Javadoc documentation on the\n`OffsetMetrics` class itself, and the methods available to you on its\nbuilder.\n\nJSONSerde\n---------\n\nA Generic JSON serializer/deserializer using Jackson for Kafka. \nThis is convenient for the common use case that JSON is being\nserialized/deserialized in Kafka.\n\nTypical usage is \n* `JsonSerde jsonSerde = JSONSerde.forType(objectMapper, class\u003cT\u003e)`. This will\nreturn an object that implements Serializer, Deserializer, and Serde interfaces.\n\nMiscellaneous\n-----\n* See the Partitions package and the Client Package. \n* A few utilities and abstractions to do with PartitioningStrategies and with a basic MessageSink/dispatcher (mostly obviated by\nEDA). These are used in buzzsaw and ab-presto-service.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopentable%2Fotj-kafka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopentable%2Fotj-kafka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopentable%2Fotj-kafka/lists"}