{"id":21179586,"url":"https://github.com/layerware/pgqueue","last_synced_at":"2025-07-09T23:31:45.625Z","repository":{"id":45240005,"uuid":"39051438","full_name":"layerware/pgqueue","owner":"layerware","description":"A Clojure durable queue implementation on top of postgresql using postgresql's advisory locks","archived":false,"fork":false,"pushed_at":"2021-12-28T21:30:41.000Z","size":123,"stargazers_count":68,"open_issues_count":2,"forks_count":7,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-24T18:06:45.911Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/layerware.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-07-14T03:28:32.000Z","updated_at":"2025-05-21T22:37:38.000Z","dependencies_parsed_at":"2022-09-06T09:20:50.756Z","dependency_job_id":null,"html_url":"https://github.com/layerware/pgqueue","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/layerware/pgqueue","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/layerware%2Fpgqueue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/layerware%2Fpgqueue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/layerware%2Fpgqueue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/layerware%2Fpgqueue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/layerware","download_url":"https://codeload.github.com/layerware/pgqueue/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/layerware%2Fpgqueue/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262597825,"owners_count":23334644,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-20T17:32:25.345Z","updated_at":"2025-07-09T23:31:45.368Z","avatar_url":"https://github.com/layerware.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pgqueue\n\nA Clojure durable queue implementation on top of postgresql using\npostgresql's advisory locks to provide non-blocking take,\nsuch that concurrent workers do not block each other.\n\n## Installation\n\n[Leiningen](https://github.com/technomancy/leiningen) dependency information:\n\n```clj\n[com.layerware/pgqueue \"0.5.1\"]\n```\n\n## Recent Changes\n\n - 0.5.1 dependency updates\n - 0.5.0 is the same API as 0.4.1, but has dependency version bumps with potentially-breaking changes:\n   - `clojure.java.jdbc 0.6.x` Ensure that your app is otherwise\n     compatible with `clojure.java.jdbc 0.6.x` breaking changes. See\n     the\n     [CHANGELOG](https://github.com/clojure/java.jdbc/blob/master/CHANGES.md). \n   - `nippy` encoding updates require that you remove/empty your\n     pgqueue queues before upgrading and re-queue these items after\n     upgrading.  If you do not need to save your current queues, you\n     can simple delete the pgqueues table for the same effect.\n - 0.4.1 fix (harmless) warning about rollback w/ autocommit on\n - 0.4.0 `pgqueue` namespace moved to `pgqueue.core` per [Clojure style guidelines](https://github.com/bbatsov/clojure-style-guide#no-single-segment-namespaces)\n - 0.3.4 Fixes reconnect bug\n - 0.3.3 Added `:analyze-threshold` option to `pgqueue.core/queue`\n\n## Usage\n\n```clj\n(require '[pgqueue.core :as pgq])\n\n; Define a queue config\n(def q-config {:db {:subprotocol \"postgresql\"\n                    :subname \"//127.0.0.1:5432/pgtest\"\n                    :user \"pgtest\"\n                    :password \"pgtest\"}})\n\n;; Create a queue with a name and config\n(def q (pgq/queue :my-queue q-config))\n\n;; Put items on the queue\n(pgq/put q :a) ;=\u003e true\n(pgq/put q :b) ;=\u003e true\n\n;; Put nil is a no-op\n(pgq/put q nil) ;=\u003e nil\n\n;; How many?\n(pgq/count q) ;=\u003e 2\n\n;; Take items off the queue\n(pgq/take q) ;=\u003e :a\n(pgq/take q) ;=\u003e :b\n(pgq/take q) ;=\u003e nil\n\n;; Safely take item from queue using pgqueue/take-with.\n;; If the work in the body crashes, the item is\n;; unlocked for other workers to take it instead.\n\n;; Successful case\n(pgq/put q :success)\n(pgq/take-with [item q]\n  ; item is locked\n  ; do work here\n  (Thread/sleep 20))\n  ; item is unlocked/deleted\n\n;; Failure case\n(pgq/put q :failme)\n(pgq/take-with [item q]\n  ; item is locked\n  (throw (ex-info \"FAIL!\" {}))) ;=\u003e ExceptionInfo FAIL!\n  \n;; it's OK, item is safe for the next taker:\n(pgq/take q) ;=\u003e :failme\n\n\n;; pgqueue is also a priority queue\n;; For put with arity of 2, a default priority of 100 is used.\n;;   (see pgqueue/queue docs to set a default priority for a queue)\n;; For put with arity of 3, the second argument is a priority integer\n;; where a lower value = higher priority; negative integers are ok\n(pgq/put q 500 \"least\")\n(pgq/put q 200 \"low\")\n(pgq/put q 100 \"medium\")\n(pgq/put q 1   \"high\")\n(pgq/put q -10 \"urgent\")\n\n(pgq/take q) ;=\u003e \"urgent\"\n(pgq/take q) ;=\u003e \"high\"\n(pgq/take q) ;=\u003e \"medium\"\n(pgq/take q) ;=\u003e \"low\"\n(pgq/take q) ;=\u003e \"least\"\n\n\n;; Put a batch\n(pgq/put-batch q [1 2 3]) ;=\u003e true\n\n;; Take a batch\n(pgq/take-batch q 3) ;=\u003e [1 2 3]\n   \n```\n\n\n## Documentation\n\n[API Docs] (http://layerware.github.io/pgqueue)\n\n\n## Why use pgqueue?\n\npgqueue is a middle-ground queueing solution between file-based\ndurable queues and larger queue/messaging servers.\n\nIf you are already using postgresql for your database and don't\nneed a heavy-load queueing solution, pgqueue might be the right fit.\n\nBenefits of pgqueue include:\n - concurrent, non-blocking take across multiple processes\n   and across multiple JVM threads\n - safely take an item for work (pgqueue/take-with) and\n   ensure that item is unlocked in the event of a crash/exception\n - backing up your postgresql database backs up your queues, too!\n\n\n## Performance\n\nObviously, a queue implementation on a relational database is going\nto perform more slowly than a dedicated queuing server.  That said,\npgqueue's use of postgresql's advisory locks is likely faster than\na strategy that locks a row for update.\n\nSee the perf project in this repository to run a performance test\non your hardware.\n\nHere are the results from a Linode 2048 \n  (2GB RAM, SSD, 2 x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz)\nusing the following JVM options in project.clj:\n```\njvm-opts ^:replace [\"-Xmx1g\" \"-Xms1g\" \"-server\"]\n```\n\n```\nPGQUEUE_CONFIG=./perf.config.edn lein run -m pgqueue.perf\npgqueue perf test\n\n       put     100 integers...    483ms duration   0.207/ms (   207/s) avg rate\n      take     100 integers...    495ms duration   0.202/ms (   202/s) avg rate\n put-batch     100 integers...     51ms duration   1.961/ms (  1961/s) avg rate\ntake-batch     100 integers...    192ms duration   0.521/ms (   521/s) avg rate\n\n       put    1000 integers...   1265ms duration   0.791/ms (   791/s) avg rate\n      take    1000 integers...   3171ms duration   0.315/ms (   315/s) avg rate\n put-batch    1000 integers...    428ms duration   2.336/ms (  2336/s) avg rate\ntake-batch    1000 integers...   1779ms duration   0.562/ms (   562/s) avg rate\n\n       put   10000 integers...   9312ms duration   1.074/ms (  1074/s) avg rate\n      take   10000 integers...  33683ms duration   0.297/ms (   297/s) avg rate\n put-batch   10000 integers...   2419ms duration   4.134/ms (  4134/s) avg rate\ntake-batch   10000 integers...  20792ms duration   0.481/ms (   481/s) avg rate\n```\n\n\n## License\n\nCopyright © 2018 Layerware, Inc.\n\nDistributed under the [Apache License, Version 2.0] (http://www.apache.org/licenses/LICENSE-2.0.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flayerware%2Fpgqueue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flayerware%2Fpgqueue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flayerware%2Fpgqueue/lists"}