{"id":20768152,"url":"https://github.com/softwaremill/kmq","last_synced_at":"2025-05-16T04:03:44.332Z","repository":{"id":37271607,"uuid":"81654110","full_name":"softwaremill/kmq","owner":"softwaremill","description":"Kafka-based message queue","archived":false,"fork":false,"pushed_at":"2025-03-25T00:07:37.000Z","size":462,"stargazers_count":337,"open_issues_count":28,"forks_count":45,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-04-08T14:08:33.421Z","etag":null,"topics":["cats-effect","java","kafka","message-queue","reactive-streams","scala"],"latest_commit_sha":null,"homepage":"https://softwaremill.com/open-source/","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/softwaremill.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-11T13:18:12.000Z","updated_at":"2025-03-25T05:09:30.000Z","dependencies_parsed_at":"2025-02-24T12:06:45.838Z","dependency_job_id":"5c00dc1c-2605-4966-a3ad-613fdea1c19a","html_url":"https://github.com/softwaremill/kmq","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkmq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkmq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkmq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/softwaremill%2Fkmq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/softwaremill","download_url":"https://codeload.github.com/softwaremill/kmq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254464891,"owners_count":22075570,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cats-effect","java","kafka","message-queue","reactive-streams","scala"],"created_at":"2024-11-17T11:35:57.121Z","updated_at":"2025-05-16T04:03:44.301Z","avatar_url":"https://github.com/softwaremill.png","language":"Scala","readme":"# Kafka Message Queue\n\n[![Join the chat at https://gitter.im/softwaremill/kmq](https://badges.gitter.im/softwaremill/kmq.svg)](https://gitter.im/softwaremill/kmq?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.softwaremill.kmq/core_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.softwaremill.kmq/core_2.12)\n\nUsing `kmq` you can acknowledge processing of individual messages in Kafka, and have unacknowledged messages \nre-delivered after a timeout. \n\nThis is in contrast to the usual Kafka offset-committing mechanism, using which you can acknowledge all messages\nup to a given offset only. \n\nIf you are familiar with [Amazon SQS](https://aws.amazon.com/sqs/), `kmq` implements a similar message processing\nmodel.\n\n# How does this work?\n\nFor a more in-depth overview see the blog: [Using Kafka as a message queue](https://softwaremill.com/using-kafka-as-a-message-queue/),\nand for performance benchmarks: [Kafka with selective acknowledgments (kmq) performance \u0026 latency benchmark](https://softwaremill.com/kafka-with-selective-acknowledgments-performance/)\n\nThe acknowledgment mechanism uses a `marker` topic, which should have the same number of partitions as the \"main\"\ndata topic (called the `queue` topic). The marker topic is used to track which messages have been processed, by \nwriting start/end  markers for every message.\n\n![message flow diagram](https://github.com/softwaremill/kmq/blob/master/kmq.png?raw=true)\n\n# Using kmq\n\nAn application using `kmq` should consist of the following components:\n\n* a number of `RedeliveryTracker`s. This components consumes the `marker` topic and redelivers messages if appropriate. \nMultiple copies should be started in a cluster for fail-over. Uses automatic partition assignment.\n* components which send data to the `queue` topic to be processed\n* queue clients, either custom or using the `KmqClient`     \n\n# Maven/SBT dependency\n\nSBT:\n\n    \"com.softwaremill.kmq\" %% \"core\" % \"0.3.1\"\n\nMaven:\n\n    \u003cdependency\u003e\n        \u003cgroupId\u003ecom.softwaremill.kmq\u003c/groupId\u003e\n        \u003cartifactId\u003ecore_2.13\u003c/artifactId\u003e\n        \u003cversion\u003e0.3.1\u003c/version\u003e\n    \u003c/dependency\u003e\n\nNote: The supported Scala versions are: 2.12, 2.13.\n\n# Client flow\n\nThe flow of processing a message is as follows:\n\n1. read messages from the `queue` topic, in batches\n2. write a `start` marker to the `markers` topic for each message, wait until the markers are written\n3. commit the biggest message offset to the `queue` topic\n4. process messages\n5. for each message, write an `end` marker. No need to wait until the markers are written.\n\nThis ensures at-least-once processing of each message. Note that the acknowledgment of each message (writing the \n`end` marker) can be done for each message separately, out-of-order, from a different thread, server or application.\n\n# Example code\n\nThere are three example applications:\n\n* `example-java/embedded`: a single java application that starts all three components (sender, client, redelivery tracker)\n* `example-java/standalone`: three separate runnable classes to start the different components\n* `example-scala`: an implementation of the client using [reactive-kafka](https://github.com/akka/reactive-kafka)\n\n# Time \u0026 timestamps\n\nHow time is handled is crucial for message redelivery, as messages are redelivered after a given amount of time passes\nsince the `start` marker was sent.\n\nTo track what was sent when, `kmq` uses Kafka's message timestamp. By default, this is messages create time\n(`message.timestamp.type=CreateTime`), but for the `markers` topic, it is advisable to switch this to `LogAppendTime`.\nThat way, the timestamps more closely reflect when the markers are really written to the log, and are guaranteed to be\nmonotonic in each partition (which is important for redelivery - see below).\n\nTo calculate which messages should be redelivered, we need to know the value of \"now\", to check which `start` markers\nhave been sent later than the configured timeout. When a marker has been received from a partition recently, the\nmaximum such timestamp is used as the value of \"now\" - as it indicates exactly how far we are in processing the\npartition. What \"recently\" means depends on the `useNowForRedeliverDespiteNoMarkerSeenForMs` config setting. Otherwise,\nthe current system time is used, as we assume that all markers from the partition have been processed.\n\n# Dead letter queue (DMQ)\nThe redelivery of the message is attempted only a configured number of times. By default, it's 3. You can change that number by setting `maxRedeliveryCount` value in `KmqConfig`.\nAfter that number is exceeded messages will be forwarded to a topic working as a *dead letter queue*. By default, the name of that topic is name of the message topic concatenated with the suffix `__undelivered`. You can configure the name by setting `deadLetterTopic` in `KmqConfig`.\nThe number of redeliveries is tracked by `kmq` with a special header. The default the name of that header is `kmq-redelivery-count`. You can change it by setting `redeliveryCountHeader` in `KmqConfig`. ","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaremill%2Fkmq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsoftwaremill%2Fkmq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsoftwaremill%2Fkmq/lists"}