{"id":21497148,"url":"https://github.com/financial-times/message-queue-consumer","last_synced_at":"2025-03-17T12:24:28.536Z","repository":{"id":4825599,"uuid":"42855774","full_name":"Financial-Times/message-queue-consumer","owner":"Financial-Times","description":"A library that is an intermediary to Confluent Kafka REST Proxy","archived":false,"fork":false,"pushed_at":"2023-11-15T08:37:46.000Z","size":115,"stargazers_count":0,"open_issues_count":17,"forks_count":1,"subscribers_count":35,"default_branch":"master","last_synced_at":"2024-11-23T16:24:08.877Z","etag":null,"topics":["confluent-kafka","kafka","rest-proxy"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Financial-Times.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-09-21T09:11:00.000Z","updated_at":"2023-05-01T03:56:34.000Z","dependencies_parsed_at":"2024-11-23T16:33:42.017Z","dependency_job_id":null,"html_url":"https://github.com/Financial-Times/message-queue-consumer","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Financial-Times%2Fmessage-queue-consumer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Financial-Times%2Fmessage-queue-consumer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Financial-Times%2Fmessage-queue-consumer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Financial-Times%2Fmessage-queue-consumer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Financial-Times","download_url":"https://codeload.github.com/Financial-Times/message-queue-consumer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235356162,"owners_count":18976823,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["confluent-kafka","kafka","rest-proxy"],"created_at":"2024-11-23T16:21:50.011Z","updated_at":"2025-01-23T22:13:32.829Z","avatar_url":"https://github.com/Financial-Times.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# message-queue-consumer\n\nA library that is an intermediary to Confluent Kafka REST Proxy. It implements v2 of the API: \u003chttps://docs.confluent.io/current/kafka-rest/api.html#api-v2\u003e.\n\nIt supports the following settings:\n\n\n```\ntopicName - name of the topic from which to consume messages\ngroupName - consumer group\nqueueProxyHost - location of the http proxy(eg \"http://localhost:8082\")\nqueue - used for dynamic routing. This values will be passed on as an http Host header on all the requests\nbackoffPeriod - period in milliseconds for which the app will sleep before trying to consume messages \n        - backoff is applied when queue is empty(last consume request returned no messages) or exception occurred when trying to connect to the proxy\nautoCommit - boolean flag which configures autoCommit when consuming messages. If true offsets are committed if the consume request to the proxy returns 200.\n           - if false offsets are manually committed after the batch of messages are processed\n           - because of the proxy limitations the recommendations are to use autocommit true for topics with small messages\noffsetReset - possible values are \"none\", \"earliest\" and \"latest\" (any other value causes a Kafka proxy error)\n            - earliest: start processing all messages from the beginning of a Kafka topic\n            - latest: start processing messages which are produced after the consumer started\n            - none: throw exception to the consumer if no previous offset is found for the consumer's group\n            - because of the proxy limitations the recommendations are to use \"latest\" unless you have a very good reasons not to\n            - using \"earliest\" will impact the memory usage of the proxy\n            - if not explicitly set the default value is \"latest\"\nstreamCount - number of threads to use for processing messages\n            - each thread will create a new proxy consumer instance which will be assigned to different kafka partition(s)\n```\nThe library expects a jersey http client to be passed in. Make sure the client you provide supports overriding http Host header.\n\n## How to use\n\n1. Add library as a dependency\n1. Configure consumer with the params explained above\n1. Add implementation for: `com.ft.message.consumer.MessageListener`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinancial-times%2Fmessage-queue-consumer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffinancial-times%2Fmessage-queue-consumer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffinancial-times%2Fmessage-queue-consumer/lists"}