{"id":15650537,"url":"https://github.com/oliyh/carmine-streams","last_synced_at":"2026-03-07T12:02:23.949Z","repository":{"id":44462507,"uuid":"243765586","full_name":"oliyh/carmine-streams","owner":"oliyh","description":"Utility functions for working with Redis streams in carmine","archived":false,"fork":false,"pushed_at":"2023-11-02T22:28:18.000Z","size":75,"stargazers_count":37,"open_issues_count":2,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-20T16:51:47.320Z","etag":null,"topics":["carmine","clojure","redis","redis-client","redis-streams","streams"],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oliyh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["oliyh"]}},"created_at":"2020-02-28T13:12:40.000Z","updated_at":"2024-11-19T21:30:45.000Z","dependencies_parsed_at":"2024-10-03T12:35:53.832Z","dependency_job_id":"0d844ae7-2ab4-4959-8cc1-08b6956edd8c","html_url":"https://github.com/oliyh/carmine-streams","commit_stats":{"total_commits":48,"total_committers":3,"mean_commits":16.0,"dds":0.04166666666666663,"last_synced_commit":"98c7e6f685ecf9171d43a2ff26555d20014029f2"},"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/oliyh/carmine-streams","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fcarmine-streams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fcarmine-streams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fcarmine-streams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fcarmine-streams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oliyh","download_url":"https://codeload.github.com/oliyh/carmine-streams/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fcarmine-streams/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30212485,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-07T09:02:10.694Z","status":"ssl_error","status_checked_at":"2026-03-07T09:02:08.429Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["carmine","clojure","redis","redis-client","redis-streams","streams"],"created_at":"2024-10-03T12:34:59.682Z","updated_at":"2026-03-07T12:02:18.922Z","avatar_url":"https://github.com/oliyh.png","language":"Clojure","funding_links":["https://github.com/sponsors/oliyh"],"categories":[],"sub_categories":[],"readme":"# carmine-streams\n\nUtility functions for working with [Redis streams](https://redis.io/topics/streams-intro) in Clojure using [carmine](https://github.com/ptaoussanis/carmine).\n\nRedis does a brilliant job of being fast with loads of features and Carmine does a brilliant job of exposing all the low-level Redis commands\nin Clojure. Working with Redis' streams API requires quite a lot of interaction to produce desirable high-level behaviour, and that is what this\nlibrary provides.\n\n**carmine-streams** allows you to create streams and consumer groups, consume streams reliably, deal with failed consumers and unprocessable messages\nand gain visibility on the state of it all with a few simple functions. A single consumer can also process messages from multiple streams in priority order.\n\n[![Clojars Project](https://img.shields.io/clojars/v/carmine-streams.svg)](https://clojars.org/carmine-streams)\n\n## Upgrade notice\n\n:fire: Version `0.2.0` was recently released with breaking API changes. Please read the [Upgrade](UPGRADE.md) guide for more information.\n\n## Usage\n\n- [Consumer groups and consumers](#consumer-groups-and-consumers)\n- [Visibility](#visibility)\n- [Recovering from failures](#recovering-from-failures)\n- [Utilities](#utilities)\n\n### Consumer groups and consumers\n\n#### Naming things\n\nConsistent naming conventions for streams, groups and consumers:\n\n```clj\n(require '[carmine-streams.core :as cs])\n(def conn-opts {})\n\n(def stream (cs/stream-name \"sensor-readings\"))        ;; -\u003e stream/sensor-readings\n(def group (cs/group-name \"persist-readings\"))         ;; -\u003e group/persist-readings\n(def consumer (cs/consumer-name \"persist-readings\" 0)) ;; -\u003e consumer/persist-readings/0\n```\n\n#### Writing to streams\n\nA convenience function `xadd-map` for writing Clojure maps to streams:\n\n```clj\n(car/wcar conn-opts (cs/xadd-map (cs/stream-name \"maps\") \"*\" {:foo \"bar\"}))\n```\n\nand parsing them back with `kvs-\u003emap`:\n\n```clj\n(let [[[_stream messages]] (car/wcar conn-opts (car/xread :count 1 :streams (cs/stream-name \"maps\") \"0-0\"))]\n  (map (fn [[_id kvs]] (cs/kvs-\u003emap kvs))\n       messages))\n\n;; [{:foo \"bar\"}]\n```\n\n#### Consumer group creation\n\nIdempotent consumer group creation:\n\n```clj\n(cs/create-consumer-group! conn-opts stream group)\n```\n\nOr create a consumer group on multiple streams at once:\n\n```clj\n(cs/create-consumer-group! conn-opts [stream1 stream2 stream3] group)\n```\n\nThis function also de-registers idle consumers on the group. The amount of time before a consumer is considered idle can be configured:\n\n\n```clj\n(cs/create-consumer-group! conn-opts stream group \"$\" {:deregister-idle (* 5 60 1000)})\n```\n\n#### Consumer creation\n\nStart an infinite loop that consumes from the group:\n\n```clj\n(def opts {:block 5000\n           :control-fn cs/default-control-fn\n           :claim-opts {:min-idle-time (* 60 1000)\n                        :max-deliveries 10\n                        :message-rescue-count 100\n                        :dlq {:stream (cs/stream-name \"dlq\")\n                              :include-message? true}}})\n\n(def consumer\n  (Thread.\n   (fn []\n     (cs/start-multi-consumer! conn-opts\n                               stream\n                               group\n                               consumer\n                               #(println \"Yum yum, tasty message\" %)\n                               opts))))\n\n(.start consumer)\n```\n\nN.B. carmine-streams does not have any opinion on threads, but it is recommended to avoid Clojure's `future` as it is unsuited to\nlong-running tasks that do not return values (like carmine-stream's consumers).\n\nConsumer behaviour when there is only one stream is as follows:\n\n - Calls the callback for every message received, with the message\n   coerced into a keywordized map, and acks the message.\n   If the callback throws an exception the message will not be acked\n - Processes all pending messages on startup before processing new ones\n - Processes new messages until either:\n   - The consumer is unblocked (see `unblock-consumers!`)\n   - There are no messages delivered during the time it was blocked\n     waiting for a new message. If this happens, it will check for\n     pending messages and begin processing the backlog if any are\n     found, returning to wait for new messages when the backlog is\n     cleared.\n\n\nWhen checking for pending messages, if it has been sufficiently long\nsince the last check, it will check for idle messages on the backlog\nof other consumers and claim them, or putting messages on the dlq if\nthey have been retried too many times. This ensures that even if a\nconsumer dies, its messages will still be processed.\n\n#### Consumers with multiple streams\n\nA consumer can also be passed multiple streams:\n\n```clj\n(def opts {:block 5000\n           :control-fn cs/default-control-fn})\n\n(def consumer\n  (Thread.\n   (fn []\n     (cs/start-multi-consumer! conn-opts\n                               [stream1 stream2 stream3]\n                               group\n                               consumer\n                               #(println \"Yum yum, tasty message\" %)\n                               opts))))\n\n(.start consumer)\n```\n\nWhen passed multiple streams, the consumer will behave similarly to\nwhen it is passed a single stream, except it will process messages\nfrom the first stream, then the second stream, then the third, etc. If\na new message arrives on a higher priority stream while it is\nreceiving messages on a lower priority stream, it will process the\nhigher priority message as soon as it has finished processing its\ncurrent message.\n\nOptions to the consumer consist of:\n- `:block` ms to block waiting for a new message when there are no\n  pending messages on any of the streams\n- `:control-fn` a function for controlling the flow of operation, see `default-control-fn`\n- `:claim-opts` an options map for configuring how messages are\n  claimed from other consumers. See [Recovering from failures](#recovering-from-failures) for available options.\n\nEach stream being processed by a multi-stream consumer will be processed as shown in this flowchart:\n![consumer flowchart](/doc/consumer-state-machine.svg)\n\n#### Control flow\n\nThe default control flow is as follows:\n- Exit on errors reading from Redis (including unblocking)\n- Recur on successful message callback\n- Recur on failed message callback\n\nYou can provide your own `:control-fn` callback to change or add additional behaviour\nto the consumer. The `control-fn` may do whatever it pleases\nbut must return either `:exit` or `:recur`. See `default-control-fn` for an example.\n\n#### Stopping consumers\n\nYou should first interrupt the threads that your consumers are running on.\nThe interrupt will be checked before each read operation and the consumer will exit gracefully.\n\n```clj\n(.interrupt consumer)\n```\n\nIn addition you should send an unblock message. This will allow the consumer to stop any blocking\nread of redis it might currently be performing in order to exit.\n\nSending an unblock message to blocked consumers can be done like this:\n\n```clj\n;; unblock all consumers matching consumer/*\n(cs/unblock-consumers! conn-opts)\n\n;; unblock only consumers matching consumer/persist-readings/*\n(cs/unblock-consumers! conn-opts (cs/consumer-name \"persist-readings\"))\n\n;; unblock all consumers of group\n(cs/unblock-consumers! conn-opts stream group)\n```\n\n### Visibility\n\n#### All stream keys\n\n```clj\n;; all stream keys matching stream/*\n(cs/all-stream-keys conn-opts) ;; -\u003e #{\"stream/sensor-readings\"}\n\n;; all stream keys matching persist-*\n(cs/all-stream-keys conn-opts \"persist-*\")\n```\n\n#### All group names for a stream\n\n```clj\n(cs/group-names conn-opts stream) ;; -\u003e #{\"group/persist-readings\"}\n```\n\n#### Stats for a consumer group\n\n```clj\n(cs/group-stats conn-opts stream group)\n\n{:name \"group/my-group\",\n :consumers ({:name \"consumer/my-consumer/0\", :pending 1, :idle 102}\n             {:name \"consumer/my-consumer/1\", :pending 0, :idle 208}\n             {:name \"consumer/my-consumer/2\", :pending 0, :idle 311}),\n :pending 1,\n :last-delivered-id \"0-2\",\n :unconsumed 0}\n```\n\n### Recovering from failures\n\nLive consumers are responsible for finding pending messages from dead consumers and claiming them so that they can be processed. This functionality is included in the `start-multi-consumer!` function, which periodically checks for such messages in addition to sending undeliverable messages to a Dead Letter Queue (DLQ).\n\nWhen a message is not acknowledged by the consumer (i.e. your consumer died halfway through,\nor the callback threw an exception) it remains pending and its idle time is how long it has been\nsince it was first read.\n\nThese two possibilities are handled differently:\n\n  - `:min-idle-time` the minimum time (ms) a message has to be idle\n    before it can be claimed. Also the minimum amount of time between\n    checking for abandoned messages\n  - `:max-deliveries` the maximum number of times a message should be\n    delivered (attempted to be processed) before it is put in the dlq\n  - `:message-rescue-count` the number of message to attempt to claim\n    in one go\n  - `:dlq` dead letter queue options map. Options are:\n    - `:stream` the stream to which poison messages are added\n    - `:include-message?` set this to false if you don't want to\n      include original message content in the dlq message\n\n\n- If your consumer died and remains dead\n  - The delivery count will remain at 1 and the idle time will increase\n  - When the idle time has increased enough that it's obvious the\n    consumer can't still be processing it we want another consumer\n    that is alive to claim it.\n  - The `:min-idle-time` option in the `:claim-opts` map inside the\n    `start-multi-consumer!` options is the time necessary for a\n    consumer/message to be considered dead before its messages may be\n    claimed by another consumer. This option is also used as the\n    minimum amount of time between checking for abandoned messages.\n- If the message was bad and the worker throws an exception trying to process it\n  - It will remain in the backlog which the worker will attempt to\n    process during quiet times\n  - The appropriate entry in the delivery counts hash-map[^1] will\n    increase on each attempt\n  - When it reaches a particular value we will decide it cannot be processed and send it to a DLQ for later inspection\n  - The `:max-deliveries` key of `:claim-opts` is the number of deliveries required before the message is considered unprocessable or 'poison'.\n  - The `:dlq` option of `:claim-opts` specifies\n    - The name of the `:stream` to write the message metadata to\n    - Whether to `:include-message?` data inside the DLQ message.\n- The `:claim-opts` map also specifies the `:message-rescue-count`:\n  the number of messages to inspect from other consumers during a\n  periodic check.\n\n[^1]: When a consumer reads from multiple streams, redis's inbuilt\n    message delivery counts are no longer useful, so a separate redis\n    hash is used to store delivery counts for a consumer group. This\n    is stored under a key generated using\n    `(cs/group-name-\u003edelivery-counts-key group)`.\n\n#### Clearing pending messages\nIf you need to clear pending messages from all consumers, or a particular one, you can use one of these:\n\n```clj\n(cs/clear-pending! conn-opts stream group) ;; clears pending messages for all consumers\n\n(cs/clear-pending! conn-opts stream group \"consumer-1\") ;; clears pending messages for 'consumer-1'\n```\n\nYou may want to pair this with trimming the stream (caveat: this can result in data loss):\n```clj\n(car/wcar conn-opts (car/xtrim stream MAXLEN 0))\n```\n\n### Utilities\n\n#### Message ids\n\nGet the next smallest message id (useful for iterating through ranges as per `xrange` or `xpending`:\n\n```clj\n(cs/next-id \"0-1\") ;; -\u003e 0-2\n```\n\nGet the largest id that is smaller than this one:\n\n```clj\n(cs/prev-id \"0-2\") ;; -\u003e 0-1\n```\n\n## Development\n\nStart a normal REPL. You will need redis-server v7.0.0+ running on the default port to run the tests.\n\n[![CircleCI](https://circleci.com/gh/oliyh/carmine-streams.svg?style=svg)](https://circleci.com/gh/oliyh/carmine-streams)\n\n## License\n\nCopyright © 2020 oliyh\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary\nLicenses when the conditions for such availability set forth in the Eclipse\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\nthe Free Software Foundation, either version 2 of the License, or (at your\noption) any later version, with the GNU Classpath Exception which is available\nat https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliyh%2Fcarmine-streams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foliyh%2Fcarmine-streams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliyh%2Fcarmine-streams/lists"}