{"id":19066739,"url":"https://github.com/washingtonpost/kafka-service-interface","last_synced_at":"2025-04-28T12:46:18.134Z","repository":{"id":45523989,"uuid":"70149882","full_name":"washingtonpost/kafka-service-interface","owner":"washingtonpost","description":"Docker service that exposes Kafka producer and consumer API's","archived":false,"fork":false,"pushed_at":"2022-11-16T04:46:55.000Z","size":51,"stargazers_count":1,"open_issues_count":2,"forks_count":5,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-04-15T09:14:30.847Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/washingtonpost.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-10-06T12:01:05.000Z","updated_at":"2021-02-05T17:31:08.000Z","dependencies_parsed_at":"2022-07-16T01:48:07.781Z","dependency_job_id":null,"html_url":"https://github.com/washingtonpost/kafka-service-interface","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/washingtonpost%2Fkafka-service-interface","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/washingtonpost%2Fkafka-service-interface/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/washingtonpost%2Fkafka-service-interface/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/washingtonpost%2Fkafka-service-interface/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/washingtonpost","download_url":"https://codeload.github.com/washingtonpost/kafka-service-interface/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223772158,"owners_count":17199977,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-09T00:57:53.655Z","updated_at":"2024-11-09T00:57:54.336Z","avatar_url":"https://github.com/washingtonpost.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# kafka-service-interface\nDocker service that exposes Kafka producer and consumer API's\n\nThis service provides an easy way to add Kafka producer and consumer API's to your application.  Communication between\nthis Kafka service and other docker compose services is done through http callbacks.\n\nThe officially supported Kafka client library is in Java.  Client libraries in other languages have been written but\nsome of then are buggy and outdated.  This approach lets you easily integrate Kafka into any other language\nwhile getting the benefits of the officially supported client libraries.\n\nImplementation details:\n* Exposes Kafka publish callback url\n* Exposes one or more Kafka consumer callback urls\n* Includes failed message retries\n* Includes a dead topic and callback url\n\n## Docker Compose\n![diagram.png](diagram.png)\n\n## Callback Format\n### Kafka Publish\n```\n{\n    topic: 'topic.name',\n    key: 'unique to spread evenly between partitions',\n    message: ''\n}\n```\n### Kafka Consumer\n```\n{\n    topic: 'topic.name',\n    key: 'unique to spread evenly between partitions',\n    offset: '',\n    partition: '',\n    received_on: \u003cMilliseconds since Epoch\u003e,\n    received_on_date: 'string date representation',\n    value: ''\n}\n```\nFailed messages will include:\n```\n{\n    retries: \u003cnumber of retries\u003e,\n    last_retry_on: \u003cMilliseconds since Epoch\u003e,\n    last_retry_on_date: 'string date representation'\n}\n```\nDead messages will include:\n```\n{\n    dead_on: \u003cMilliseconds since Epoch\u003e,\n    dead_on_date: 'string date representation'\n}\n```\n\n## Samples\n\nTo see how this can be added to your application see the [samples](samples) directory.\n\n## Example Configuration\n### Docker Compose\nAdd the Kafka Service Interface service.\n```services:\n     kafkaserviceinterfacesample:\n       build: .\n       links:\n         - kafkaserviceinterface\n       environment:\n         # The sample will publish kafka messages to this callback url hosted by the Kafka Service Interface.\n         KAFKA_PUBLISH_URL: 'http://kafkaserviceinterface:8080/kafka/publish'\n         KAFKA_TOPIC: 'kafkaserviceinterfacesample.topic1'\n         KAFKA_TOPIC_TEST: 'kafkaserviceinterfacesample.test1'\n\n     kafkaserviceinterface:\n       image: quay.io/washpost/kafka-service-interface\n       volumes:\n         - ./config:/config\n       environment:\n         app.name: 'Kafka Service Interface Node.js Sample'\n         vpc: 'arc2'\n         # Configuration file used to define the Kafka publisher and consumers.\n         config.location: '/config/config.yml'\n\n   version: '2'\n```\n\n### config.yml\nThe config.yml file is used to configure the Kafka producer and consumers\n```\nkafka:\n  host: \"kafka.arc2.nile.works\"\n  port: \"9092\"\nkafka.producer:\n  # Other services will call this endpoint to publish to Kafka.  http:\u003cservice name\u003e:8080/kafka/publish\n  publish.path: \"/kafka/publish\"\n  port: 8080\n  properties:\n    client.id: \"test\"\n    acks: \"1\"\n    retries: \"5\"\n    retry.backoff.ms: \"1000\"\n    reconnect.backoff.ms: \"1000\"\n    key.serializer: \"org.apache.kafka.common.serialization.StringSerializer\"\n    value.serializer: \"org.apache.kafka.common.serialization.StringSerializer\"\n    batch.size: \"32000\"\n    linger.ms: \"10\"\nkafka.consumers:\n  -\n    topics: \"kafkaserviceinterfacesample.topic1\"\n    # For this consumer, when messages are read from Kafka, this endpoint will be called.\n    callback.url: \"http://kafkaserviceinterfacesample:3000/kafka-consumption-callback\"\n    # If the endpoint above fails the message will be put on this (optional) topic.\n    failure.topic: \"kafkaserviceinterfacesample.failure1\"\n    # Failed messages will be retried \u003cfailure.retries\u003e number of times before they are moved to the \u003cdead.topic\u003e\n    failure.retries: 3\n    # Once the message has failed \u003cfailure.retries\u003e times, it is moved to the (optional\u003e \u003cdead.topic\u003e.\n    dead.topic: \"kafkaserviceinterfacesample.dead1\"\n    # Once moved to the \u003cdead.topic\u003e the message is also sent to this (optional) callback url.\n    dead.callback.url: \"http://kafkaserviceinterfacesample:3000/dead-callback\"\n    properties:\n      group.id: \"kafkaserviceinterfacesample.test\"\n      max.partition.fetch.bytes: \"1000000\"\n      enable.auto.commit: \"false\"\n      key.deserializer: \"org.apache.kafka.common.serialization.StringDeserializer\"\n      value.deserializer: \"org.apache.kafka.common.serialization.StringDeserializer\"\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwashingtonpost%2Fkafka-service-interface","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwashingtonpost%2Fkafka-service-interface","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwashingtonpost%2Fkafka-service-interface/lists"}