{"id":15632295,"url":"https://github.com/oliyh/superlifter","last_synced_at":"2025-05-15T17:02:41.521Z","repository":{"id":49898897,"uuid":"167527996","full_name":"oliyh/superlifter","owner":"oliyh","description":"A DataLoader for Clojure/script","archived":false,"fork":false,"pushed_at":"2025-01-08T13:52:10.000Z","size":75,"stargazers_count":182,"open_issues_count":4,"forks_count":13,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-31T21:46:15.738Z","etag":null,"topics":["clojure","clojurescript","dataloader","fetch","graphql","lacinia","urania"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/oliyh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["oliyh"]}},"created_at":"2019-01-25T10:14:13.000Z","updated_at":"2025-01-09T21:42:34.000Z","dependencies_parsed_at":"2024-06-21T05:47:07.390Z","dependency_job_id":"e0e53467-5d97-4c5a-b6bc-1ce5e319c3fb","html_url":"https://github.com/oliyh/superlifter","commit_stats":{"total_commits":40,"total_committers":6,"mean_commits":6.666666666666667,"dds":"0.22499999999999998","last_synced_commit":"823883edfda26fb1d96f4321e8d6ae967b92747c"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fsuperlifter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fsuperlifter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fsuperlifter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/oliyh%2Fsuperlifter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/oliyh","download_url":"https://codeload.github.com/oliyh/superlifter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247737788,"owners_count":20987721,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","clojurescript","dataloader","fetch","graphql","lacinia","urania"],"created_at":"2024-10-03T10:43:19.775Z","updated_at":"2025-04-07T22:07:13.920Z","avatar_url":"https://github.com/oliyh.png","language":"Clojure","readme":"# superlifter\n\nSuperlifter is an implementation of [DataLoader](https://github.com/graphql/dataloader) for Clojure.\n\nTo quote from the DataLoader readme:\n\n\u003e DataLoader allows you to decouple unrelated parts of your application without sacrificing the performance of batch data-loading. While the loader presents an API that loads individual values, all concurrent requests will be coalesced and presented to your batch loading function. This allows your application to safely distribute data fetching requirements throughout your application and maintain minimal outgoing data requests.\n\nSuperlifter uses [Urania](https://github.com/funcool/urania), a remote data access library for Clojure/script inspired by [Haxl](https://github.com/facebook/Haxl)\nwhich in turn inspired DataLoader. Urania allows batching of similar fetches and deduplication via caching of identical fetches.\n\nSuperlifter adds smooth integration with libraries like [lacinia](https://github.com/walmartlabs/lacinia), where GraphQL resolvers are run independently\nand must return data (or promises of data), leading to 1+n problems which can otherwise only be resolved by prefetching which complicates code.\n\nThe aim of superlifter is to provide a way of combining fetches delineated by time buckets, thresholds or explicit trigger rather than by node resolution.\n\nAs the underlying fetches are performed by Urania, knowledge of this library is required (it's very simple, though!).\n\nSuperlifter provides the following features:\n\n- Fast, simple implementation of DataLoader pattern\n- Bucketing by time or by queue size\n- Asynchronous fetching\n- Batching of fetches\n- Shared cache for all fetches in a session\n  - Guarantees consistent results\n  - Avoids duplicating work\n- Access to the cache allows longer-term persistence\n\n[![Clojars Project](https://img.shields.io/clojars/v/superlifter.svg)](https://clojars.org/superlifter)\n\n- [Vanilla usage](#vanilla-usage)\n- [Lacinia usage](#lacinia-usage)\n\n## Vanilla usage\n\nStart a superlifter as follows:\n\n```clj\n(require '[superlifter.api :as s])\n(require '[urania.core :as u])\n\n(def context (s/start! {:buckets {:default {:triggers {}}}}))\n```\n\nThis superlifter has no triggers, and must be fetched manually.\nOther kinds of trigger include `queue-size` and `interval` (like DataLoader), detailed below.\nRemember to call `(s/stop! context)` when you have finished using it.\n\nYou can enqueue items for fetching:\n\n```clj\n(s/with-superlifter context\n  (def hello-promise (s/enqueue! (u/value \"Hello world\"))))\n```\n\nWhen the fetch is triggered the promises will be delivered.\n\n### Triggering fetches\n\nRegardless of the trigger used, you can always manually trigger a fetch of whatever is currently in the queue using\n`(s/with-superlifter context (s/fetch!))`.\nThis returns a promise which is delivered when all the fetches in the queue are complete, containing the results of all the fetches.\n\n#### On demand\n\nIn the example above no triggers were specified. Fetches will only happen when you call\n`(s/with-superlifter context (s/fetch!))`.\n\n#### Queue size trigger\n\nYou can specify that the queue is fetched when the queue reaches a certain size. You can configure this to e.g. 10 using the following options:\n```clj\n{:triggers {:queue-size {:threshold 10}}}\n```\n\n#### Elastic trigger\n\nYou can specify that the queue is fetched when the queue size exceeds the threshold. The threshold can be updated dynamically and snaps\nback to zero when a fetch is performed, in contrast to the queue size trigger which remains at a fixed size. The trigger can be specified as follows:\n```clj\n{:triggers {:elastic {:threshold 0}}}\n```\n\n#### Interval trigger\nYou can specify that the queue is fetched every e.g. 100ms using the following options:\n```clj\n{:triggers {:interval {:interval 100}}}\n```\n\nThis will give batching by time in a similar fashion to DataLoader.\n\n#### Debounced trigger\nYou can specify that the queue is fetched when no items have been added within the last e.g. 100ms with these options\n```clj\n{:triggers {:debounced {:interval 100}}}\n```\n\n#### Your own trigger\n\nYou can register your own kind of trigger by participating the in `s/start-trigger!` multimethod, so you can listen for other kinds of events that might let you know when it's a good time to perform the fetch.\nSee the interval trigger implementation for inspiration.\n\n#### Trigger combinations\nYou can supply any number of triggers which will all run concurrently and the queue will be fetched when any one condition is met.\n\n```clj\n{:triggers {:queue-size {:threshold 10}\n            :interval {:interval 100}}}\n```\n\nIt is recommended that a `:queue-size` trigger is always used in combination with an `:interval` or `debounced` trigger in order to avoid\nhanging when you have e.g. a queue size of 5 but only four muses are enqueued within it.\n\n## Lacinia usage\n\nGiven the following schema in lacinia:\n\n```clj\n(def schema\n {:objects {:PetDetails {:fields {:name {:type 'String}\n                                  :age {:type 'Int}}}\n            :Pet {:fields {:id {:type 'String}\n                           :details {:type :PetDetails\n                                     :resolve resolve-pet-details}}}}\n  :queries {:pets\n            {:type '(list :Pet)\n             :resolve resolve-pets}}})\n```\n\nWhere the resolvers are as follows:\n\n```clj\n(defn- resolve-pets [context args parent]\n  (let [ids (keys (:db context))]\n    (map (fn [id] {:id id}) ids)))\n\n;; invoked n times, once for every id from the parent resolver\n(defn- resolve-pet-details [context args {:keys [id]}]\n  (get-in (:db context) id))\n```\n\nWe can rewrite this using superlifter (see the [example code](https://github.com/oliyh/superlifter/tree/master/example) for full context):\n\n```clj\n;; superlifter.lacinia has a different `with-superlifter` macro\n;; to help you return a lacinia promise\n(require '[superlifter.lacinia :refer [with-superlifter]])\n(require '[clojure.tools.logging :as log])\n\n;; def-fetcher - a convenience macro like defrecord for things which cannot be combined\n(s/def-fetcher FetchPets []\n  (fn [_this env]\n    (map (fn [id] {:id id}) (keys (:db env)))))\n\n;; def-superfetcher - a convenience macro like defrecord for combinable things\n(s/def-superfetcher FetchPet [id]\n  (fn [many env]\n    (log/info \"Combining request for\" (count many) \"pets\")\n    (map (:db env) (map :id many))))\n\n(defn- resolve-pets [context _args _parent]\n  (with-superlifter context\n    (-\u003e (s/enqueue! (-\u003eFetchPets))\n        (s/update-trigger! :pet-details :elastic\n                           (fn [trigger-opts pet-ids]\n                             (update trigger-opts :threshold + (count pet-ids)))))))\n\n(defn- resolve-pet-details [context _args {:keys [id]}]\n  (with-superlifter context\n    (s/enqueue! :pet-details (-\u003eFetchPet id))))\n```\nNote that when defining a Superfetcher as above, the number of outputs is \nexpected to match the number of inputs, and to be in the same order. In \ncases where there may be results missing, or results might arrive in a \ndifferent order, there is a second arity of `def-superfetcher` that takes \nboth a `match-fn` and a `missing-fn` to define how results should be joined \nwith inputs.\n\nIt's usual to start a Superlifter before each query and stop it afterwards.\nThere is an `inject-superlifter` interceptor which will help you do this:\n\n```clj\n(require '[com.walmartlabs.lacinia.pedestal :as lacinia])\n(require '[com.walmartlabs.lacinia.schema :as schema])\n(require '[superlifter.lacinia :refer [inject-superlifter]])\n\n(def pet-db (atom {\"abc-123\" {:name \"Lyra\"\n                              :age 11}\n                   \"def-234\" {:name \"Pantalaimon\"\n                              :age 11}\n                   \"ghi-345\" {:name \"Iorek\"\n                              :age 41}}))\n\n(def lacinia-opts {:graphiql true})\n\n(def superlifter-args\n  {:buckets {:default {:triggers {:queue-size {:threshold 1}}}\n             :pet-details {:triggers {:elastic {:threshold 0}}}}\n   :urania-opts {:env {:db @pet-db}}})\n\n(def service\n  (lacinia/service-map\n   (fn [] (schema/compile schema))\n   (assoc lacinia-opts\n          :interceptors (into [(inject-superlifter superlifter-args)]\n                              (lacinia/default-interceptors (fn [] (schema/compile schema)) lacinia-opts)))))\n```\n\n## Development\n\n`cider-jack-in-clj\u0026cljs` and open the test page http://localhost:9500/figwheel-extra-main/auto-testing\n\n## Build\n[![CircleCI](https://circleci.com/gh/oliyh/superlifter.svg?style=svg)](https://circleci.com/gh/oliyh/superlifter)\n\n## Sponsorship\nDevelopment and maintenance is generously sponsored by @toyokumo\n\n## License\n\nCopyright © 2019 oliyh\n\nDistributed under the Eclipse Public License either version 1.0 or (at\nyour option) any later version.\n","funding_links":["https://github.com/sponsors/oliyh"],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliyh%2Fsuperlifter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foliyh%2Fsuperlifter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foliyh%2Fsuperlifter/lists"}