{"id":26186943,"url":"https://github.com/ivarref/yoltq","last_synced_at":"2025-10-06T08:59:20.359Z","repository":{"id":40771731,"uuid":"403037881","full_name":"ivarref/yoltq","owner":"ivarref","description":"An opinionated Datomic queue for building (more) reliable systems. Supports retries, backoff, ordering and more.","archived":false,"fork":false,"pushed_at":"2024-06-14T14:10:50.000Z","size":132,"stargazers_count":47,"open_issues_count":4,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T12:38:23.494Z","etag":null,"topics":["clojure","datomic","queue"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivarref.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-04T11:22:42.000Z","updated_at":"2024-11-01T14:11:40.000Z","dependencies_parsed_at":"2025-03-11T23:46:06.945Z","dependency_job_id":null,"html_url":"https://github.com/ivarref/yoltq","commit_stats":null,"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fyoltq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fyoltq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fyoltq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fyoltq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivarref","download_url":"https://codeload.github.com/ivarref/yoltq/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248985950,"owners_count":21194020,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","datomic","queue"],"created_at":"2025-03-11T23:35:44.226Z","updated_at":"2025-10-06T08:59:20.352Z","avatar_url":"https://github.com/ivarref.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# yoltq\n\nAn opinionated Datomic queue for building (more) reliable systems.\nImplements the\n[transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html)\npattern.\nSupports retries, backoff, ordering and more.\nOn-prem only.\n\n## Installation\n\n[![Clojars Project](https://img.shields.io/clojars/v/com.github.ivarref/yoltq.svg)](https://clojars.org/com.github.ivarref/yoltq)\n\n## 1-minute example\n\n```clojure\n(require '[com.github.ivarref.yoltq :as yq])\n\n(def conn (datomic.api/connect \"...\"))\n\n; Initialize system\n(yq/init! {:conn conn})\n\n; Add a queue consumer that will intentionally fail on the first attempt\n(yq/add-consumer! :q\n                 (let [cnt (atom 0)]\n                   (fn [payload]\n                     (when (= 1 (swap! cnt inc))\n                       ; A consumer throwing an exception is considered a queue job failure\n                       (throw (ex-info \"failed\" {})))\n                     ; Anything else than a throwing exception is considered a queue job success\n                     ; This includes nil, false and everything else.\n                     (log/info \"got payload\" payload))))\n\n; Start threadpool that picks up queue jobs\n(yq/start!)\n\n; Queue a job\n@(d/transact conn [(yq/put :q {:work 123})])\n\n; On your console you will see something like this:\n; 17:29:54.598 DEBUG queue item 613... for queue :q is pending status :init\n; 17:29:54.602 DEBUG queue item 613... for queue :q now has status :processing\n; 17:29:54.603 DEBUG queue item 613... for queue :q is now processing\n; 17:29:54.605 WARN  queue-item 613... for queue :q now has status :error after 1 try in 4.8 ms\n; 17:29:54.607 WARN  error message was: \"failed\" for queue-item 613...\n; 17:29:54.615 WARN  ex-data was: {} for queue-item 613...\n; The item is so far failed...\n\n; But after approximately 10 seconds have elapsed, the item will be retried:\n; 17:30:05.596 DEBUG queue item 613... for queue :q now has status :processing\n; 17:30:05.597 DEBUG queue item 613... for queue :q is now processing\n; 17:30:05.597 INFO  got payload {:work 123}\n; 17:30:05.599 INFO  queue-item 613... for queue :q now has status :done after 2 tries in 5999.3 ms\n; And then it has succeeded.\n```\n\n## Rationale\n\nIntegrating with external systems that may be unavailable can be tricky.\nImagine the following code:\n\n```clojure\n(defn post-handler [user-input]\n  (let [db-item (process user-input)\n        ext-ref (clj-http.client/post ext-service {:connection-timeout 3000 ; milliseconds \n                                                   :socket-timeout     10000 ; milliseconds\n                                                   ...})] ; may throw exception\n    @(d/transact conn [(assoc db-item :some/ext-ref ext-ref)])))\n```\n\nWhat if the POST request fails? Should it be retried? For how long?\nShould it be allowed to fail? How do you then process failures later?\n\nPS: If you do not set connection/socket-timeout, there is a chance that\nclj-http/client will wait for all eternity in the case of a dropped TCP connection.\n\nThe queue way to solve this would be:\n\n```clojure\n(defn get-ext-ref [{:keys [id]}]\n  (let [ext-ref (clj-http.client/post ext-service {:connection-timeout 3000  ; milliseconds \n                                                   :socket-timeout     10000 ; milliseconds\n                                                   ...})] ; may throw exception\n    @(d/transact conn [[:db/cas [:some/id id]\n                        :some/ext-ref\n                        nil\n                        ext-ref]])))\n\n(yq/add-consumer! :get-ext-ref get-ext-ref {:allow-cas-failure? true})\n\n(defn post-handler [user-input]\n  (let [{:some/keys [id] :as db-item} (process user-input)]\n    @(d/transact conn [db-item\n                       (yq/put :get-ext-ref {:id id})])))\n```\n\nHere `post-handler` will always succeed as long as the transaction commits.\n\n`get-ext-ref` may fail multiple times if `ext-service` is down.\nThis is fine as long as it eventually succeeds.\n\nThere is a special case where `get-ext-ref` succeeds, but \nsaving the new queue job status to the database fails.\nThus `get-ext-ref` and any queue consumer should tolerate to \nbe executed successfully several times.\n\nFor `get-ext-ref` this is solved by using\nthe database function\n[:db/cas (compare-and-swap)](https://docs.datomic.com/on-prem/transactions/transaction-functions.html#dbfn-cas)\nto achieve a write-once behaviour.\nThe yoltq system treats cas failures as job successes\nwhen a consumer has `:allow-cas-failure?` set to `true` in its options.\n\n## How it works\n\n### Queue jobs\n\nCreating queue jobs is done by `@(d/transact conn [...other data... (yq/put :q {:work 123})])`.\nInspecting `(yq/put :q {:work 123})]` you will see something like this:\n\n```clojure\n#:com.github.ivarref.yoltq{:id #uuid\"614232a8-e031-45bb-8660-be146eaa32a2\", ; Queue job id \n                           :queue-name :q, ; Destination queue                                 \n                           :status :init, ; Status\n                           :payload \"{:work 123}\", ; Payload persisted to the database with pr-str\n                           :bindings \"{}\", ; Bindings that will be applied before executing consumer function\n                           :lock #uuid\"037d7da1-5158-4243-8f72-feb1e47e15ca\", ; Lock to protect from multiple consumers\n                           :tries 0, ; How many times the job has been executed\n                           :init-time 4305758012289 ; Time of initialization (System/nanoTime)\n                           }\n```\n\nThis is the queue job as it will be stored into the database. \nYou can see that the payload, i.e. the second argument of `yq/put`,\nis persisted into the database. Thus the payload must be `pr-str`-able (unless you have specified\ncustom `:encode` and `:decode` functions that override this).\n\n\nA queue job will initially have status `:init`.\nIt will then transition to the following statuses:\n\n* `:processing`: When the queue job begins processing in the queue consumer function.\n* `:done`: If the queue consumer function returns normally.\n* `:error`: If the queue consumer function throws an exception.\n\n### Queue consumers\n\nQueue jobs will be consumed by queue consumers. A consumer is a function taking a single argument,\nthe payload. It can be added like this:\n\n```clojure\n(yq/add-consumer! \n  :q ; Queue to consume  \n  (fn [payload] (println \"got payload:\" payload)) ; Queue consumer function\n  ; An optional map of queue opts\n  {:allow-cas-failure? true ; Treat [:db.cas ...] failures as success. This is one way for the\n                            ; consumer function to ensure idempotence.\n   :valid-payload? (fn [payload] (some? (:id payload))) ; Function that verifies payload.\n                                                        ; Should return truthy for valid payloads.\n                                                        ; The default function always returns true.\n   :max-retries 10})        ; Specify maximum number of times an item will be retried. Default: 10000.\n                            ; If :max-retries is given as 0, the job will ~always be retried, i.e.\n                            ; 9223372036854775807 times (Long/MAX_VALUE).\n```\n\nThe `payload` will be deserialized from the database using `clojure.edn/read-string` before\ninvocation, i.e. you will get back what you put into `yq/put`.\n\nThe yoltq system treats a queue consumer function invocation as successful if it does not throw\nan exception. Any return value, be it `nil`, `false`, `true`, etc. is considered a success.\n\n### Listening for queue jobs\n\nWhen `(yq/start!)` is invoked, a threadpool is started.\n\nOne thread is permanently allocated for listening to the \n[tx-report-queue](https://docs.datomic.com/on-prem/clojure/index.html#datomic.api/tx-report-queue)\nand responding to changes. This means that yoltq will respond \nand process newly created queue jobs fairly quickly.\nThis also means that queue jobs in status `:init` will almost always be processed without\nany type of backoff.\n\nThe threadpool also schedules polling jobs that will check for various statuses regularly:\n\n* Jobs in status `:error` that have waited for at least `:error-backoff-time` (default: 5 seconds) will be retried.\n* Jobs that have been in `:processing` for at least `:hung-backoff-time` (default: 30 minutes) will be considered hung and retried.\n* Old `:init-backoff-time` (default: 1 minute) `:init` jobs that have not been processed. Queue jobs can be left in status `:init` during application restart/upgrade, and thus the need for this strategy.\n\n\n### Retry and backoff strategy\n\nYoltq assumes that if a queue consumer throws an exception for one item, it\nwill also do the same for another item in the immediate future, \nassuming the remote system that the queue consumer represents is still down.\nThus if there are ten failures for queue `:q`, it does not make sense to\nretry all of them at once.\n\nThe retry polling job that runs regularly (`:poll-delay`, default: every 10 seconds)\nthus stops at the first failure.\nEach queue have their own polling job, so if one queue is down, it will *not* stop\nother queues from retrying.\n\nThe retry polling job will continue to eagerly process queue jobs as long as it \nencounters only successes.\n\nWhile the `:error-backoff-time` of default 5 seconds may seem short, in practice\nif there is a lot of failed items and the external system is still down,\nthe actual backoff time will be longer.\n\n\n### Stuck threads and stale jobs\n\nA single thread is dedicated to monitoring how much time a queue consumer \nspends on a single job. If this exceeds `:max-execute-time` (default: 5 minutes)\nthe stack trace of the offending consumer will be logged as `:ERROR`.\n\nIf a job is found stale, that is if the database spent time exceeds \n`:hung-backoff-time` (default: 30 minutes),\nthe job will either be retried or marked as `:error`. This case may happen if the application\nis shut down abruptly during processing of queue jobs.\n\n\n### Giving up\n\nA queue job will remain in status `:error` once `:max-retries` (default: 10000) have been reached.\nIf `:max-retries` is given as `0`, the job will be retried 9223372036854775807 times before\ngiving up.\nIdeally this should not happen. ¯\\\\\\_(ツ)\\_/¯\n\n### Custom encoding and decoding\n\nYoltq will use `pr-str` and `clojure.edn/read-string` by default to encode and decode data.\nYou may specify `:encode` and `:decode` either globally or per queue to override this behaviour.\nThe `:encode` function must return a byte array or a string.\n\nFor example if you want to use [nippy](https://github.com/ptaoussanis/nippy):\n```clojure\n(require '[taoensso.nippy :as nippy])\n\n; Globally for all queues:\n(yq/init!\n  {:conn conn\n   :encode nippy/freeze\n   :decode nippy/thaw})\n\n; Or per queue:\n(yq/add-consumer! \n  :q ; Queue to consume  \n  (fn [payload] (println \"got payload:\" payload)) ; Queue consumer function\n  {:encode nippy/freeze\n   :decode nippy/thaw}) ; Queue options, here with :encode and :decode\n```\n\n### Partitions\n\nYoltq supports specifying which\n[partition](https://docs.datomic.com/on-prem/schema/schema.html#partitions)\nqueue entities should belong to.\nThe default function is:\n```clojure\n(defn default-partition-fn [_queue-name]\n  (keyword \"yoltq\" (str \"queue_\" (.getValue (java.time.Year/now)))))\n```\nThis is to say that there will be a single partition per year for yoltq.\nYoltq will take care of creating the partition if it does not exist.\n\nYou may override this function, either globally or per queue, with the keyword `:partition-fn`.\nE.g.:\n```clojure\n(yq/init! {:conn conn :partition-fn (fn [_queue-name] :my-partition)})\n```\n\n### All configuration options\n\nFor an exhaustive list of all configuration options,\nsee\n[yq/default-opts](https://github.com/ivarref/yoltq/blob/main/src/com/github/ivarref/yoltq.clj#L21).\n\n# Groups of Jobs\n\nYoltq supports grouping jobs in a queue, and tracking the progress of such a\ngroup of jobs. Consider this example: your system is used by the marketing\ndepartment to send emails to groups of users. Multiple colleagues in the\nmarketing department could potentially do this at the same time, but they want\nto see the progress of their _own_ campagne, not that of _all_ emails being\nsent. When adding the jobs to the queue, you can specify the `job-group`\nparameter, in this case indicate which marketeer is running the jobs:\n\n```clojure\n(doseq [uid user-ids]\n  @(d/transact conn [(yq/put :send-mail\n                             ; Payload:\n                             {:user-id uid :from ... :to ... :body ...}\n                             ; Job options:\n                             {:job-group :mail-campagne/for-marketeer-42})]))\n```\n\nWhen you want to know the progress of that specific job group, and display it in\nyour user interface, you can use `job-group-progress`, which returns a structure\nsimilar to `queue-stats`:\n\n```clojure\n(yq/job-group-progress :send-mail :mail-campagne/for-marketeer-42)\n;; =\u003e [{:qname :send-mail\n;;      :job-group :mail-campagne/for-marketeer-42\n;;      :status :init\n;;      :count 78}\n;;     {:qname :send-mail\n;;      :job-group :mail-campagne/for-marketeer-42\n;;      :status :done\n;;      :count 24}]\n```\n\n# Regular and REPL usage\n\nFor a regular system and/or REPL session you'll want to do:\n\n```clojure\n(require '[com.github.ivarref.yoltq :as yq])\n\n(yq/init! {:conn conn})\n\n(yq/add-consumer! :q-one ...)\n(yq/add-consumer! :q-two ...)\n\n; Start yoltq system\n(yq/start!)\n\n; Oops I need another consumer. This works fine:\n(yq/add-consumer! :q-three ...)\n\n; When the application is shutting down:\n(yq/stop!)\n```\n\nYou may invoke `yq/add-consumer!` and `yq/init!` on a live system as you like.\n\nIf you change `:pool-size` or `:poll-delay` you will have to `(yq/stop!)` and\n`(yq/start!)` to make changes take effect.\n\n## Queue job dependencies and ordering\n\nIt is possible to specify that one queue job must wait for another queue\njob to complete before it will be executed:\n\n```clojure\n@(d/transact conn [(yq/put :a \n                           ; Payload:\n                           {:id \"a1\"}\n                           ; Job options:\n                           {:id \"a1\"})])\n\n@(d/transact conn [(yq/put :b \n                           ; Payload:\n                           {:id \"b1\" :a-ref \"a1\"}\n                           ; Jobs options:\n                           {:depends-on [:a \"a1\"]})])\n\n; depends-on may also be specified as a function of the payload when \n; adding the consumer:\n(yq/add-consumer! :b \n                  (fn [payload] ...)\n                  {:depends-on (fn [payload]\n                                 [:a (:a-ref payload)])})\n```\n\nHere queue job `b1` will not execute before `a1` is `:done`.\n\nNote that queue-name plus `:id` in job options must be an unique value.\nIn the example above that means `:a` plus `a1` must be unique.\n\nWhen specifying `:depends-on`, the referred job must at least exist in the database,\notherwise `yq/put` will throw an exception.\n\nOther than this there is no attempt at ordering the execution of queue jobs.\nIn fact the opposite is done in the poller to guard against the case that a single failing queue job\ncould effectively take down the entire retry polling job.\n\n## Retrying jobs in the REPL\n\n```clojure\n(require '[com.github.ivarref.yoltq :as yq])\n\n; List jobs that are in state error:\n(yq/get-errors :q)\n\n; This will retry a single job that is in error, regardless \n; of how many times it has been retried earlier.\n; If the job fails, you will get the full stacktrace on the REPL.\n(yq/retry-one-error! :q)\n; Returns a map containing the new state of the job.\n; Returns nil if there are no (more) jobs in state error for this queue.\n```\n\n# Testing\n\nFor testing you will probably want determinism over an extra threadpool\nby using the test queue:\n\n```clojure\n...\n(:require [clojure.test :refer :all]\n  [com.github.ivarref.yoltq :as yq]\n  [com.github.ivarref.yoltq.test-queue :as tq])\n\n; Enables the test queue and disables the threadpool for each test.\n; yq/start! and yq/stop! becomes a no-op.\n(use-fixtures :each tq/call-with-virtual-queue!)\n\n(deftest demo\n         (let [conn ...]\n           (yq/init! {:conn conn}) ; Setup\n           (yq/add-consumer! :q identity)\n\n           @(d/transact conn [(yq/put :q {:work 123})]) ; Add work\n\n           ; tq/consume! consumes one job and asserts that it succeeds.\n           ; It returns the return value of the consumer function\n           (is (= {:work 123} (tq/consume! :q)))\n           \n           ; If you want to test the idempotence of your function, \n           ; you may force retry a consumer function:\n           ; This may for example be useful to verify that the\n           ; :db.cas logic is correct.\n           (is (= {:work 123} (tq/force-retry! :q)))))\n```\n\n## Logging and capturing bindings\n\nYoltq can capture and restore dynamic bindings.\nIt will capture during `yq/put` and restore them when the consumer function\nis invoked. This is specified in the `:capture-bindings` setting.\nIt defaults to `['#taoensso.timbre/*context*]`, \ni.e. the [timbre](https://github.com/ptaoussanis/timbre) log context,\nif available, otherwise an empty vector.\n\nThese dynamic bindings will be in place when yoltq logs errors, warnings\netc. about failing consumer functions, possibly making troubleshooting\neasier.\n\n## Limitations\n\nDatomic does not have anything like `for update skip locked`.\nThus consuming a queue should be limited to a single JVM process.\nThis library will take queue jobs by compare-and-swapping a lock+state,\nprocess the item and then compare-and-swapping the lock+new-state.\nIt does so eagerly, thus if you have multiple JVM consumers you will\nmost likely get many locking conflicts. It should work, but it's far\nfrom optimal.\n\n## Alternatives\n\nI did not find any alternatives for Datomic.\n\nIf I were using PostgreSQL or any other database that supports\n`for update skip locked`, I'd use a queue that uses this.\nFor Clojure there is [proletarian](https://github.com/msolli/proletarian).\n\nFor Redis there is [carmine](https://github.com/ptaoussanis/carmine).\n\nNote: I have not tried these libraries myself.\n\n## Other stuff\n\nIf you liked this library, you may also like:\n\n* [conformity](https://github.com/avescodes/conformity):\n  A Clojure/Datomic library for idempotently transacting norms into your database – be they schema,\n  data, or otherwise.\n* [datomic-schema](https://github.com/ivarref/datomic-schema):\n  Simplified writing of Datomic schemas (works with conformity).\n* [double-trouble](https://github.com/ivarref/double-trouble):\n  Handle duplicate Datomic transactions with ease.\n* [gen-fn](https://github.com/ivarref/gen-fn):\n  Generate Datomic function literals from regular Clojure namespaces.\n* [rewriting-history](https://github.com/ivarref/rewriting-history):\n  A library to rewrite Datomic history.\n\n## Change log\n\n#### [Unreleased]\n\n#### [0.2.94] - 2025-09-22\n\nAdded support for [groups of jobs](#groups-of-jobs).\nThanks [Stefan van den Oord](https://github.com/svdo)!\n\n#### [0.2.85] - 2025-07-29\n\nSame as v0.2.82, but without the `v` prefix.\n\n#### [v0.2.82] - 2025-06-18\n\nAdded support for specifying `tx-report-queue` as a keyword in `init!`. Yoltq will\nthen not grab the datomic report queue, but use the one provided:\n\n```clojure\n(require '[com.github.ivarref.yoltq :as yq])\n(yq/init! {:conn            conn\n           :tx-report-queue (yq/get-tx-report-queue-multicast! conn :yoltq)\n                            ; ^^ can be any `java.util.concurrent.BlockingQueue` value\n                            })\n\n(another-tx-report-consumer! (yq/get-tx-report-queue-multicast! conn :another-consumer-id))\n\n```\n\nAdded multicast support for `datomic.api/tx-report-queue`:\n```clojure\n(require '[com.github.ivarref.yoltq :as yq])\n(def my-q1 (yq/get-tx-report-queue-multicast! conn :q-id-1))\n; ^^ consume my-q1 just like you would do `datomic.api/tx-report-queue`\n\n(def my-q2 (yq/get-tx-report-queue-multicast! conn :q-id-2))\n; Both my-q1 and my-q2 will receive everything from `datomic.api/tx-report-queue`\n; for the given `conn`\n\n(def my-q3 (yq/get-tx-report-queue-multicast! conn :q-id-3 true))\n; my-q3 sets the optional third argument, `send-end-token?`, to true.\n; The queue will then receive `:end` if the queue is stopped.\n; This can enable simpler consuming of queues:\n(future\n  (loop []\n    (let [q-item (.take ^java.util.concurrent.BlockingQueue my-q3)]\n      (if (= q-item :end)\n        (println \"Time to exit. Goodbye!\")\n        (do\n          (println \"Processing q-item\" q-item)\n          (recur))))))\n\n; The default value for `send-end-token?` is `false`, i.e. the behaviour will be\n; identical to that of datomic.api/tx-report-queue.\n\n@(d/transact conn [{:db/doc \"new-data\"}])\n\n; Stop the queue:\n(yq/stop-multicast-consumer-id! conn :q-id-3)\n=\u003e true\n; The multicaster thread will send `:end`.\n; The consumer thread will then print \"Time to exit. Goodbye!\".\n\n; if the queue is already stopped (or never was started), the `stop-multicaster...` \n; functions will return false:\n(yq/stop-multicast-consumer-id! conn :already-stopped-queue-or-typo)\n=\u003e false\n\n; Stop all queues for all connections:\n(yq/stop-all-multicasters!)\n```\n\n`yq/get-tx-report-queue-multicast!` returns, like\n`datomic.api/tx-report-queue`,\n`java.util.concurrent.BlockingQueue` and starts a background thread that does \nthe multicasting as needed. Identical calls to `yq/get-tx-report-queue-multicast!`\nreturns the same `BlockingQueue`.\n\nChanged the default for `max-retries` from `10000` to `9223372036854775807`.\n\nFixed reflection warnings.\n\n#### 2023-03-20 v0.2.64 [diff](https://github.com/ivarref/yoltq/compare/v0.2.63...v0.2.64)\n\nAdded support for `max-retries` being `0`, meaning the job should be retried forever\n(or at least 9223372036854775807 times).\n\nChanged the default for `max-retries` from `100` to `10000`.\n\n#### 2022-11-18 v0.2.63 [diff](https://github.com/ivarref/yoltq/compare/v0.2.62...v0.2.63)\nAdded custom `:encode` and `:decode` support.\n\nAdded support for specifying `:partifion-fn` to specify\nwhich partition a queue item should belong to.\nIt defaults to:\n```clojure\n(defn default-partition-fn [_queue-name]\n  (keyword \"yoltq\" (str \"queue_\" (.getValue (Year/now)))))\n```\nYoltq takes care of creating the partition if it does not exist.\n\n#### 2022-11-15 v0.2.62 [diff](https://github.com/ivarref/yoltq/compare/v0.2.61...v0.2.62)\nAdded function `processing-time-stats`:\n\n```clojure\n(ns com.github.ivarref.yoltq)\n\n(defn processing-time-stats\n  \"Gather processing time statistics.\n\n  Optional keyword arguments:\n  * :age-days —  last number of days to look at data from. Defaults to 30.\n                 Use nil to have no limit.\n\n  * :queue-name — only gather statistics for this queue name. Defaults to nil, meaning all queues.\n\n  * :duration-\u003elong - Specify what unit should be used for values.\n                      Must take a java.time.Duration as input and return a long.\n\n                      Defaults to (fn [duration] (.toSeconds duration).\n                      I.e. the default unit is seconds.\n\n  Example return value:\n  {:queue-a {:avg 1\n             :max 10\n             :min 0\n             :p50 ...\n             :p90 ...\n             :p95 ...\n             :p99 ...}}\"\n [{:keys [age-days queue-name now db duration-\u003elong]\n  :or   {age-days 30\n         now      (ZonedDateTime/now ZoneOffset/UTC)\n         duration-\u003elong (fn [duration] (.toSeconds duration))}}]\n  ...)\n```\n\n#### 2022-09-07 v0.2.61 [diff](https://github.com/ivarref/yoltq/compare/v0.2.60...v0.2.61)\nAdded function `retry-stats`:\n\n```clojure\n(ns com.github.ivarref.yoltq)\n\n(defn retry-stats\n  \"Gather retry statistics.\n\n  Optional keyword arguments:\n  * :age-days —  last number of days to look at data from. Defaults to 30.\n  * :queue-name — only gather statistics for this queue name. Defaults to nil, meaning all queues.\n\n  Example return value:\n  {:queue-a {:ok 100, :retries 2, :retry-percentage 2.0}\n   :queue-b {:ok 100, :retries 75, :retry-percentage 75.0}}\n\n  From the example value above, we can see that :queue-b fails at a much higher rate than :queue-a.\n  Assuming that the queue consumers are correctly implemented, this means that the service representing :queue-b\n  is much more unstable than the one representing :queue-a. This again implies\n  that you will probably want to fix the downstream service of :queue-b, if that is possible.\n  \"\n  [{:keys [age-days queue-name now]\n    :or   {age-days 30\n           now      (ZonedDateTime/now ZoneOffset/UTC)}}]\n  ...)\n```\n\n#### 2022-08-18 v0.2.60 [diff](https://github.com/ivarref/yoltq/compare/v0.2.59...v0.2.60)\nImproved: Added config option `:healthy-allowed-error-time`:\n```\n    ; If you are dealing with a flaky downstream service, you may not want\n    ; yoltq to mark itself as unhealthy on the first failure encounter with\n    ; the downstream service. Change this setting to let yoltq mark itself\n    ; as healthy even though a queue item has been failing for some time.\n    :healthy-allowed-error-time    (Duration/ofMinutes 15)\n```\n\n#### 2022-08-15 v0.2.59 [diff](https://github.com/ivarref/yoltq/compare/v0.2.58...v0.2.59)\nFixed:\n* Race condition that made the following possible: `stop!` would terminate the slow thread \nwatcher, and a stuck thread could keep `stop!` from completing! \n\n#### 2022-06-30 v0.2.58 [diff](https://github.com/ivarref/yoltq/compare/v0.2.57...v0.2.58)\nSlightly more safe EDN printing and parsing.\nRecommended reading:\n[Pitfalls and bumps in Clojure's Extensible Data Notation (EDN)](https://nitor.com/en/articles/pitfalls-and-bumps-clojures-extensible-data-notation-edn)\n\n#### 2022-06-29 v0.2.57 [diff](https://github.com/ivarref/yoltq/compare/v0.2.56...v0.2.57)\nAdded `(get-errors qname)` and `(retry-one-error! qname)`.\n\nImproved:\n`unhealthy?` will return `false` for the first 10 minutes of the application lifetime.\nThis was done in order to push new code while a queue was in error in an earlier\nversion of the code. In this way rolling upgrades are possible regardless if there\nare queue errors.\nCan you tell that this issue hit me? ¯\\\\\\_(ツ)\\_/¯\n\n#### 2022-06-22 v0.2.56 [diff](https://github.com/ivarref/yoltq/compare/v0.2.55...v0.2.56)\nAdded support for `:yoltq/queue-id` metadata on functions. I.e. it's possible to write\nthe following:\n```clojure\n(defn my-consumer\n  {:yoltq/queue-id :some-queue}\n  [payload]\n  :work-work-work)\n\n(yq/add-consumer! #'my-consumer ; \u003c-- will resolve to :some-queue \n                  my-consumer)\n\n@(d/transact conn [(yq/put #'my-consumer ; \u003c-- will resolve to :some-queue\n                           {:id \"a\"})])\n```\n\nThe idea here is that it is simpler to jump to var definitions than going via keywords,\nwhich essentially refers to a var/function anyway. \n\n#### 2022-03-29 v0.2.55 [diff](https://github.com/ivarref/yoltq/compare/v0.2.54...v0.2.55)\nAdded: `unhealthy?` function which returns `true` if there are queues in error,\nor `false` otherwise.\n\n#### 2022-03-28 v0.2.54 [diff](https://github.com/ivarref/yoltq/compare/v0.2.51...v0.2.54)\nFixed: Schedules should now be using milliseconds and not nanoseconds.\n\n#### 2022-03-28 v0.2.51 [diff](https://github.com/ivarref/yoltq/compare/v0.2.48...v0.2.51)\n* Don't OOM on migrating large amounts of data. \n* Respect `:auto-migrate? false`.\n\n#### 2022-03-27 v0.2.48 [diff](https://github.com/ivarref/yoltq/compare/v0.2.46...v0.2.48)\n* Auto migration is done in the background.\n* Only poll for current version of jobs, thus no races for auto migration.\n\n#### 2022-03-27 v0.2.46 [diff](https://github.com/ivarref/yoltq/compare/v0.2.41...v0.2.46)\n* Critical bugfix that in some cases can lead to stalled jobs.\n```\nStarted using (System/currentTimeMillis) and not (System/nanoTime)\nwhen storing time in the database. \n```\n\n* Bump Clojure to `1.11.0`.\n\n#### 2022-03-27 v0.2.41 [diff](https://github.com/ivarref/yoltq/compare/v0.2.39...v0.2.41)\n* Added function `healthy?` that returns:\n```\n  true if no errors\n  false if one or more errors\n  nil if error-poller is yet to be executed.\n```\n\n* Added default functions for `:on-system-error` and `:on-system-recovery`\n  that simply logs that the system is in error (ERROR level) or has \n  recovered (INFO level). \n  \n* Added function `queue-stats` that returns a nicely \"formatted\"\n  vector of queue stats, for example:\n```\n  (queue-stats)\n  =\u003e\n  [{:qname :add-message-thread, :status :done, :count 10274}\n   {:qname :add-message-thread, :status :init, :count 30}\n   {:qname :add-message-thread, :status :processing, :count 1}\n   {:qname :send-message, :status :done, :count 21106}\n   {:qname :send-message, :status :init, :count 56}]\n```\n\n#### 2021-09-27 v0.2.39 [diff](https://github.com/ivarref/yoltq/compare/v0.2.37...v0.2.39)\nAdded `:valid-payload?` option for queue consumers.\n\n#### 2021-09-27 v0.2.37 [diff](https://github.com/ivarref/yoltq/compare/v0.2.33...v0.2.37) \nImproved error reporting.\n\n#### 2021-09-24 v0.2.33\n\nFirst publicly announced release.\n\n## Making a new release\n\nGo to https://github.com/ivarref/yoltq/actions/workflows/release.yml and press `Run workflow`.\n\n## License\n\nCopyright © 2021-2022 Ivar Refsdal\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary\nLicenses when the conditions for such availability set forth in the Eclipse\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\nthe Free Software Foundation, either version 2 of the License, or (at your\noption) any later version, with the GNU Classpath Exception which is available\nat https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivarref%2Fyoltq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivarref%2Fyoltq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivarref%2Fyoltq/lists"}