{"id":26186940,"url":"https://github.com/ivarref/clj-paginate","last_synced_at":"2025-10-17T21:36:03.647Z","repository":{"id":59775034,"uuid":"436298771","full_name":"ivarref/clj-paginate","owner":"ivarref","description":"Fast pagination of vectors and maps with Clojure for GraphQL.","archived":false,"fork":false,"pushed_at":"2022-10-06T08:41:53.000Z","size":78,"stargazers_count":20,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-30T07:57:48.192Z","etag":null,"topics":["clojure","graphql","paginate","pagination","paginator"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ivarref.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-12-08T15:33:14.000Z","updated_at":"2024-06-22T15:21:47.000Z","dependencies_parsed_at":"2022-09-21T14:25:38.486Z","dependency_job_id":null,"html_url":"https://github.com/ivarref/clj-paginate","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/ivarref/clj-paginate","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fclj-paginate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fclj-paginate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fclj-paginate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fclj-paginate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ivarref","download_url":"https://codeload.github.com/ivarref/clj-paginate/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ivarref%2Fclj-paginate/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266101171,"owners_count":23876733,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","graphql","paginate","pagination","paginator"],"created_at":"2025-03-11T23:35:43.154Z","updated_at":"2025-10-17T21:36:03.566Z","avatar_url":"https://github.com/ivarref.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# clj-paginate\n\nA Clojure (JVM only) implementation of the \n[GraphQL Cursor Connections Specification](https://relay.dev/graphql/connections.htm) \nwith vector or map as the backing data.\n\nSupports: \n* Collections that grows and/or changes.\n* Long polling (`:first` only, not `:last`).\n* Multiple sort criteria.\n* Ascending or descending sorting.\n* Basic OR filtering (maps only).\n* Batching (optional).\n\nNo external dependencies.\n\n## Prerequisites\n\nThe user of this library is assumed to be moderately\nfamiliar with [GraphQL pagination](https://graphql.org/learn/pagination/)\nand know the basic structure of the \n[GraphQL Cursor Connections Specification](https://relay.dev/graphql/connections.htm),\nparticularly the fact that the desired response looks like the following:\n\n```\n{\"edges\": [{\"node\": ..., \"cursor\": ...},\n           {\"node\": ..., \"cursor\": ...},\n           {\"node\": ..., \"cursor\": ...},\n           ...] \n \"pageInfo\": {\"hasNextPage\":  Boolean\n              \"hasPrevPage\":  Boolean\n              \"totalCount\":   Integer\n              \"startCursor\":  String\n              \"endCursor\":    String}}\n```\n\n## Installation\n\n[![Clojars Project](https://img.shields.io/clojars/v/com.github.ivarref/clj-paginate.svg)](https://clojars.org/com.github.ivarref/clj-paginate)\n\n## 1-minute example\n\n```clojure\n(require '[com.github.ivarref.clj-paginate :as cp])\n\n(defn nodes [page]\n  (-\u003e\u003e page\n       :edges\n       (mapv :node)))\n\n(def data (vec (shuffle [{:inst 0}\n                         {:inst 1}\n                         {:inst 2}])))\n\n; Get the initial page:\n(def page-1 (cp/paginate \n               data\n               \n               ; The next argument, `sort-attrs`, specifies how the vector should be sorted. \n               ; It must be a single keyword, a vector of keywords or a vector of pairs (keyword and :asc/:desc).\n               ; See more documentation below for information about ascending or descending\n               ; sorting.\n               :inst\n               \n               ; A function to transform an initial node into a final node,\n               ; i.e. load more data from a database.\n               identity\n               \n               ; What to get, the first two elements in this case:\n               {:first 2}))\n; page-1\n;=\u003e\n;{:edges\n; [{:node {:inst 0}, :cursor \"{:context {} :cursor [0 ]}\"}\n;  {:node {:inst 1}, :cursor \"{:context {} :cursor [1 ]}\"}],\n; :pageInfo\n; {:hasPrevPage false,\n;  :hasNextPage true,\n;  :startCursor \"{:context {} :cursor [0 ]}\",\n;  :endCursor \"{:context {} :cursor [1 ]}\",\n;  :totalCount 3}}\n\n\n; Get the second page:\n(def page-2 (cp/paginate data\n                         :inst\n                         identity\n                         {:first 2\n                          :after (get-in page-1 [:pageInfo :endCursor])}))\n; (nodes page-2)\n; =\u003e [{:inst 2}]\n\n; Get the next (empty) page:\n(def page-3 (cp/paginate data\n                         :inst\n                         identity\n                         {:first 2\n                          :after (get-in page-2 [:pageInfo :endCursor])}))\n; (nodes page-3)\n; =\u003e []\n; No more data! \n; The poller, i.e. a different backend, should now sleep for some time before attempting again.\n\n\n; More data has arrived:\n(def data [{:inst 0} ; old\n           {:inst 1} ; old\n           {:inst 2} ; old\n           {:inst 3} ; new item\n           {:inst 4} ; new item\n           ])\n\n; Time for another poll. Growing data is handled:\n(def page-4 (cp/paginate data\n                         :inst\n                         identity\n                         {:first 2\n                          :after (get-in page-3 [:pageInfo :endCursor])}))\n; (nodes page-4)\n; =\u003e [{:inst 3} {:inst 4}]\n\n; More data has arrived, and old data expired/got removed:\n(def data [{:inst 6}\n           {:inst 7}\n           {:inst 8}])\n\n; Changed data is handled as long as the newer data adheres to sorting\n(def page-5 (cp/paginate data\n                         :inst\n                         identity\n                         {:first 2\n                          :after (get-in page-4 [:pageInfo :endCursor])}))\n; (nodes page-5)\n; =\u003e [{:inst 6} {:inst 7}]\n```\n\n## Background\n\nThis library was developed for supporting pagination for \"heavy\" Datomic queries that\nspent too much time on delivering the initial result that would then have to be sorted and\npaginated.\n\n## Data requirements\n\nNodes must be maps.\n\n## Basic use case example\n\n```clojure\n(require '[com.github.ivarref.clj-paginate :as cp])\n\n(def data\n  (vec (shuffle [{:inst 0 :id 1}\n                 {:inst 1 :id 2}\n                 {:inst 2 :id 3}])))\n\n(defn http-post-handler \n  [response data http-body]\n  (assoc response\n    :status 200\n    :body (cp/paginate\n            ; The first argument is the data to paginate.\n            data\n            \n            ; The second argument specifies how the vector is sorted.\n            ; It thus also specifies what constitute a unique identifier for a node.\n            ; It may be a single keyword, a vector of keywords,\n            ; or a vector of pairs where each pair has a keyword and :asc or :desc.\n            [:inst]\n            \n            ; The third argument is a function that further processes the node.\n            ; The function may for example load more data from a database or other external storage.\n            (fn [{:keys [inst id] :as node}]  \n              (Thread/sleep 10) ; Do some heavy work.\n              (assoc node :value-from-db 1))\n            \n            ; The fourth argument should be a map containing the arguments to the pagination.\n            ; This map must contain either:\n            ; :first (Integer), how many items to fetch from the start, and optionally :after, the cursor,\n            ; or :last (Integer), how many items to fetch from the end, and optionally :before, the cursor.\n            ; If this requirement is not satisfied, an exception will be thrown.\n            http-body)))\n```\n\nThat is all that is needed for the basic use case to work.\n\n## Multiple sort criteria and descending values example\n\nThe default behaviour of `clj-paginate` is to assume that all attributes in `:sort-attrs` is sorted ascendingly.\nIt's possible to override this behaviour using pairs of `keyword :asc/:desc` in the `:sort-attrs` vector:\n\n```clojure\n(require '[com.github.ivarref.clj-paginate :as cp])\n\n(def data\n  (vec (shuffle [{:inst #inst\"2000\" :id 1}\n                 {:inst #inst\"2000\" :id 2}\n                 {:inst #inst\"2001\" :id 3}])))\n\n(cp/paginate\n   ; The first argument is the data to paginate.\n   data\n   \n   ; The second argument specifies how the vector should be sorted.\n   [[:inst :asc] [:id :desc]]\n   \n   identity\n   {:first 2})\n\n(def conn *1)\n\n(mapv :node (:edges conn))\n=\u003e [{:inst #inst\"2000\", :id 2} {:inst #inst\"2000\", :id 1}]\n\n(cp/paginate\n   ; The first argument is the data to paginate.\n   ; The data must already be sorted.\n   data\n\n   ; The second argument specifies which attributes constitute a unique identifier for a node.\n   ; It may be a single keyword, or a vector of keywords.\n   [[:inst :asc] [:id :desc]]\n\n   identity\n   {:first 2 :after (get-in conn [:pageInfo :endCursor])})\n\n(mapv :node (:edges *1))\n=\u003e [{:inst #inst\"2001\", :id 3}]\n```\n\n## OR filters\n\nSometimes you may want to provide filtering of the data.\nThis is done in two steps:\n\n1. Your HTTP endpoint must support a parameter that represents the or filter.\n2. Pass a map to the `paginate` function along with `:filter` in `opts`.\n   `:filter` should be a vector of the keys of the map that you want to filter on. \n\nAs an example, let's add a `:status` property to our previous example and make it filterable:\n\n```clojure\n(require '[com.github.ivarref.clj-paginate :as cp])\n\n(def data\n   (group-by :status\n             [{:inst 0 :id 1 :status :init}\n              {:inst 1 :id 2 :status :pending}\n              {:inst 2 :id 3 :status :done}\n              {:inst 3 :id 4 :status :error}\n              {:inst 4 :id 5 :status :done}]))\n\n(defn http-post-handler\n   [response data http-body]\n   (assoc response\n      :status 200\n      :body (cp/paginate\n               data ; data is now a map.\n               :inst\n               (fn [{:keys [inst id] :as node}]\n                  (Thread/sleep 10) ; Do some heavy work.\n                  (assoc node :value-from-db 1))\n\n               ; Assume that the HTTP endpoint accepts a parameter `:statuses` for the body,\n               ; and that when present, this is a vector such as `[:init :pending :done :error]` or similar,\n               ; i.e. the keys of `data` that we want to filter on.\n               ;\n               ; Paginate's `opts` accepts a key `:filter` that does exactly this for data maps.\n               ; Thus we can simply rename `:statuses` to `:filter` in the http body.\n               ; clj-paginate takes care of storing the value of `:filter` in the cursor\n               ; for subsequent queries.\n               (clojure.set/rename-keys http-body {:statuses :filter}))))\n\n; To illustrate this, consider the following code:\n(let [conn (cp/paginate\n              data\n              :inst\n              identity\n              {:first  1\n               :filter [:done]})]\n   ; Will print [{:inst 2, :id 3, :status :done}].\n   (println (mapv :node (:edges conn))) \n\n   ; Will print [{:inst 4, :id 5, :status :done}].\n   ; Notice here that we do not re-specify `:filter`.\n   ; It is already stored in the cursor from the original connection.\n   (println (mapv :node (:edges (cp/paginate\n                                   data\n                                   :inst\n                                   identity\n                                   {:first 1\n                                    :after (get-in conn [:pageInfo :endCursor])})))))\n```\n\nThe consumer client only needs to send `:statuses` on the initial query.\nWhen subsequent iteration is done, the cursor, `:after` or `:before`,\nalready includes `:filter`, and thus it is not necessary to re-send\nthis information on every request. If `:filter` is not specified for\na map, every key is included.\n\n### Refresh a page\n\nIf you want to refresh a page, you may add `:inclusive? true` as\na named parameter when calling `paginate`.\nThe results will then include the given cursor. This is useful\nif you want to check for updates of a given page only based on a\nprevious pageInfo.\n\n```clojure\n(require '[com.github.ivarref.clj-paginate :as cp])\n\n; Using :first:\n(cp/paginate\n  data\n  [:sort-attrs]\n  identity\n  {:first 10 :after (:startCursor (:pageInfo connection))}\n  :inclusive? true)\n\n; Using :last:\n(cp/paginate\n   data\n   [:sort-attrs]\n   identity\n   {:last 10 :before (:endCursor (:pageInfo connection))}\n   :inclusive? true)\n```\n\n### Batching\n\nBatching is supported. Add `:batch? true` when calling `paginate`.\n`f`, the third parameter to paginate, must now accept a vector of nodes, and return \na vector of processed nodes. The returned vector must have the same\nordering as the input vector.  You may want to use the function\n`ensure-order` to make sure the order is correct:\n\n```clojure\n(require '[com.github.ivarref.clj-paginate :as cp])\n\n(defn load-batch [nodes]\n  (let [loaded-nodes (-\u003e\u003e (mapv :id nodes)\n                          \n                          ; load data from database using pull-many:\n                          (datomic.api/pull-many datomic-db '[:*])\n\n                          ; Do we have any ordering guarantees? Pretend the ordering got mixed up:\n                          (shuffle))]\n    (cp/ensure-order nodes \n                     loaded-nodes\n                     :sf :id ; Source id function, defaults to :id.\n                     :df :db/id ; Dest id function, defaults to :id.\n                     \n                     ; (sf input-node) must be equal to some (df output-node).\n                     ; ensure-order uses this to order `loaded-nodes` according\n                     ; to how `nodes` were ordered.\n                     )))\n\n; Using load-batch\n(cp/paginate\n   data\n   :id\n   load-batch\n   {:first 100}\n   \n   ; The named parameter :batch? is set to `true`:\n   :batch? true\n   )\n```\n\n\n## Performance\n\n`clj-paginate` treats the (sorted) input vectors as binary trees,\nand thus the general performance is `O(log n)` for finding where to continue\ngiving out data. When paginating over maps, this\nhas to be multiplied by the number of selected keys.\n\nUsing `:first 1000` and 10 million dummy entries, the average\noverhead was about 1-5 ms per iteration on my machine. That is about\n1-5 microsecond per returned node.\n\n## Change log\n\n### 2022-10-06 0.3.54\n\nAdd support for `inclusive?`, multiple sort criteria with `:asc` or `:desc`.\nAdded named parameter `sort?` which defaults to `true`.\n\n### 2022-09-23 0.2.53\nBugfix. Values for `pageInfo.hasPrevPage` and `pageInfo.hasNextPage` for `last/before` pagination were reversed. Thanks [@kthu](https://github.com/kthu)!\n\n### 2022-09-20 0.2.52\nSupport descending values.\n\n### 2022-02-16 0.2.51\nInitial release publicly announced.\n\n## Misc\n\nA few days after I made the initial announcement, I came across\n[java.util.NavigableSet](https://docs.oracle.com/javase/8/docs/api/java/util/NavigableSet.html)\nthat looks like a perfect fit for doing pagination\nin JVM-land.\n\n## License\n\nCopyright © 2022 Ivar Refsdal\n\nThis program and the accompanying materials are made available under the\nterms of the Eclipse Public License 2.0 which is available at\nhttp://www.eclipse.org/legal/epl-2.0.\n\nThis Source Code may also be made available under the following Secondary\nLicenses when the conditions for such availability set forth in the Eclipse\nPublic License, v. 2.0 are satisfied: GNU General Public License as published by\nthe Free Software Foundation, either version 2 of the License, or (at your\noption) any later version, with the GNU Classpath Exception which is available\nat https://www.gnu.org/software/classpath/license.html.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivarref%2Fclj-paginate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fivarref%2Fclj-paginate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fivarref%2Fclj-paginate/lists"}