{"id":19917197,"url":"https://github.com/athos/drains","last_synced_at":"2025-05-03T06:30:47.333Z","repository":{"id":62432452,"uuid":"136042886","full_name":"athos/Drains","owner":"athos","description":"A new abstraction for flexible and efficient sequence aggregation in Clojure(Script)","archived":false,"fork":false,"pushed_at":"2018-08-08T12:16:35.000Z","size":116,"stargazers_count":34,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-07T11:51:29.353Z","etag":null,"topics":["clojure","clojurescript","transducers"],"latest_commit_sha":null,"homepage":"","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/athos.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-06-04T15:08:26.000Z","updated_at":"2024-05-31T07:57:18.000Z","dependencies_parsed_at":"2022-11-01T21:00:44.159Z","dependency_job_id":null,"html_url":"https://github.com/athos/Drains","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/athos%2FDrains","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/athos%2FDrains/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/athos%2FDrains/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/athos%2FDrains/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/athos","download_url":"https://codeload.github.com/athos/Drains/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252154732,"owners_count":21702982,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","clojurescript","transducers"],"created_at":"2024-11-12T21:49:07.579Z","updated_at":"2025-05-03T06:30:46.838Z","avatar_url":"https://github.com/athos.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Drains\n[![Clojars Project](https://img.shields.io/clojars/v/drains.svg)](https://clojars.org/drains)\n[![CircleCI](https://circleci.com/gh/athos/Drains.svg?style=shield)](https://circleci.com/gh/athos/Drains)\n[![codecov](https://codecov.io/gh/athos/Drains/branch/master/graph/badge.svg)](https://codecov.io/gh/athos/Drains)\n\nDrains: A new abstraction for flexible and efficient sequence aggregation in Clojure(Script)\n\nA drain is a stateful object that consists of a reducing fn and an accumulated value. Drains can be used as composable and reusable building blocks for constructing sequence aggregation.\nThis library provides several easy ways to combining multiple drains and to produce a new drain from another one, and also provides a couple of custom aggregation functions such as `reduce`, `reductions` and `fold`.\n\n## SYNOPSIS\n\n```clj\n(require '[drains.core :as d])\n\n(def countries\n  [{:name \"Canada\" :area 9984 :population 36 :continent \"North America\"}\n   {:name \"China\" :area 9634 :population 1390 :continent \"Asia\"}\n   {:name \"Germany\" :area 357 :population 82 :continent \"Europe\"}\n   {:name \"Japan\" :area 377 :population 126 :continent \"Asia\"}\n   {:name \"UK\" :area 244 :population 63 :continent \"Europe\"}\n   {:name \"USA\" :area 9628 :population 327 :continent \"North America\"}])\n\n(def sum (d/drain +))\n(def total-population (d/with (map :population) sum))\n(def total-area (d/with (map :area) sum))\n\n(d/reduce (d/drains {:total-population total-population\n                     :total-area total-area\n                     :continents (d/group-by :continent\n                                             (d/drains {:countries (d/drain (map :name) conj [])\n                                                        :population total-population\n                                                        :area total-area}))})\n          countries)\n;=\u003e {:total-population 2024,\n;    :total-area 30224,\n;    :continents {\"North America\" {:countries [\"Canada\" \"USA\"], :population 363, :area 19612},\n;                 \"Asia\" {:countries [\"China\" \"Japan\"], :population 1516, :area 10011},\n;                 \"Europe\" {:countries [\"Germany\" \"UK\"], :population 145, :area 601}}}\n```\n\n## Table of contents\n\n- [Installation](#installation)\n- [Usage](#usage)\n    - [`d/drain`, `d/reduce`](#ddrain-dreduce)\n    - [`d/drains`](#ddrains)\n    - [`d/fmap`, `d/combine-with`](#dfmap-dcombine-with)\n    - [`d/group-by`](#dgroup-by)\n    - [`d/with`](#dwith)\n    - [`d/reductions`, `d/fold`](#dreductions-dfold)\n- [Performance](#performance)\n- [Related works](#related-works)\n\n## Installation\n\nAdd the following to your `:dependencies`:\n\n[![Clojars Project](https://clojars.org/drains/latest-version.svg)](https://clojars.org/drains)\n\nIf you would rather use an unstable version of the library via [Clojure CLI tool](https://clojure.org/guides/deps_and_cli), add the following to your `deps.edn` instead:\n\n```clj\n{...\n :deps {...\n        athos/drains {:git/url \"https://github.com/athos/Drains.git\"\n                      :sha \"\u003ccommit sha\u003e\"}\n        ...}\n ...}\n```\n\n## Usage\n\nTo use this library, first require the `drains.core` ns:\n\n```clj\n(require '[drains.core :as d])\n```\n\n### `d/drain`, `d/reduce`\n\n`d/drain` creates a new drain object consisting of a reducing fn and an initial value:\n\n```clj\n(d/drain + 0)\n```\n\nOr shortly, you can just write `(d/drain +)` if the reducing fn can generate the initial value when called with no arguments.\n\nUsing the `d/reduce`, you can aggregate a sequence like `clojure.core/reduce`:\n\n```clj\n(d/reduce (d/drain conj []) (range 5))\n;=\u003e [0 1 2 3 4]\n\n(d/reduce (d/drain +) (range 5))\n;=\u003e 10\n```\n\n`d/drain` optionally takes a [transducer](https://clojure.org/reference/transducers), in which case the transducer will be applied to the reducing fn:\n\n```clj\n(d/reduce (d/drain (map inc) conj [])\n          (range 5))\n;=\u003e [1 2 3 4 5]\n\n(d/reduce (d/drain (comp (filter even?) (take 3)) conj [])\n          (range 10))\n;=\u003e [0 2 4]\n```\n\nIn general, `(d/reduce (d/drain xf op val) xs)` is semantically equal to `(transduce xf op val xs)`.\n\nNote that the reducing fn passed to a drain must have an arity-1 signature for cooperating with transducers. If your reducing fn can't be called with one argument, use `clojure.core/completing` to adding an arity-1 signature to it:\n\n```clj\n;; You don't have to wrap + with clojure.core/completing since + has an arity-1 signature\n(d/reduce (d/drain + 0) [3 1 4 1 5])\n;=\u003e 14\n\n;; But the reducing fn below doesn't, so it's necessary to be wrapped with clojure.core/completing\n(d/reduce (d/drain (fn [n d] (+ (* 10 n) d)) 0) [3 1 4 1 5])\n;; ArityException Wrong number of args (1) passed to: user/eval1108/fn--1109\n(d/reduce (d/drain (completing (fn [n d] (+ (* 10 n) d))) 0) [3 1 4 1 5])\n;=\u003e 31415\n```\n\n### `d/drains`\n\nAn interesting nature of drains is that they can be composed surprisingly easily. `d/drains` is the simplest way to compose existing drains:\n\n```clj\n(d/reduce (d/drains [(d/drain conj)\n                     (d/drain +)])\n          (range 5))\n;=\u003e [[0 1 2 3 4] 10]\n\n(d/reduce (d/drains [(d/drain min ##Inf)\n                     (d/drain max ##-Inf)])\n          [3 1 4 1 5 9 2])\n;=\u003e [1 9]\n\n;; d/drains can also take a map, not only a vector\n(d/reduce (d/drains {:min (d/drain min ##Inf)\n                     :max (d/drain max ##-Inf)})\n          [3 1 4 1 5 9 2])\n;=\u003e {:min 1, :max 9}\n\n(d/reduce (d/drains {:evens (d/drain (filter even?) conj [])\n                     :odds (d/drain (filter odd?) conj [])})\n          (range 10))\n;=\u003e {:evens [0 2 4 6 8], :odds [1 3 5 7 9]}\n\n(d/reduce (d/drains {:sum (d/drain +)\n                     :range (d/drains {:min (d/drain min ##Inf)\n                                       :max (d/drain max ##-Inf)})})\n          [3 1 4 1 5 9 2])\n;=\u003e {:sum 25, :range {:min 1, :max 9}}\n```\n\n`d/drains` can compose an arbitrary number of drains and can also be nested arbitrarily. Even in such cases, the sequence aggregation will be done in a single pass.\n\n### `d/fmap`, `d/combine-with`\n\n`d/fmap` is another way to create a drain from another drain. It can transform the form of the aggregation result:\n\n```clj\n(d/reduce (d/fmap (fn [sum] {:sum sum})\n                  (d/drain + 0))\n          (range 10))\n;=\u003e {:sum 45}\n\n(d/reduce (d/fmap (fn [[sum count]] {:average (/ sum (double count))})\n                  (d/drains [(d/drain + 0)\n                             (d/drain (map (constantly 1)) + 0)]))\n          (range 10))\n;=\u003e {:average 4.5}\n```\n\nThe combination of `d/drains` and `d/fmap` is useful and relatively common, so Drains provides the alias for that: `d/combine-with`. With `d/combine-with`, you can rewrite the above example like the following:\n\n```clj\n(d/reduce (d/combine-with (fn [sum count] {:average (/ sum (double count))})\n                          (d/drain + 0)\n                          (d/drain (map (constantly 1)) + 0))\n          (range 10))\n```\n\n### `d/group-by`\n\nAnother convenient facility is `d/group-by`. `d/group-by` creates a fresh copy of the given drain every time it encounters a new key value (calculated with the specified key-fn), and manages each of the copies respectively through the aggregation:\n\n```clj\n(d/reduce (d/group-by even? (d/drain conj))\n          (range 10))\n;=\u003e {true [0 2 4 6 8], false [1 3 5 7 9]}\n\n(d/reduce (d/group-by #(rem % 3)\n                      (d/drains {:items (d/drains conj)\n                                 :sum (d/drain +)}))\n          (range 10))\n;=\u003e {0 {:items [0 3 6 9], :sum 18},\n;    1 {:items [1 4 7], :sum 12},\n;    2 {:items [2 5 8], :sum 15}}\n```\n\n### `d/with`\n\nYou can also attach a transducer to existing drains using `d/with`:\n\n```clj\n(d/reduce (d/with (map :x)\n                  (d/drains [(d/drain (filter even?) conj [])\n                             (d/drain (filter odd?) conj [])]))\n          [{:x 1} {:x 2} {:x 3} {:x 4} {:x 5}])\n;=\u003e [[2 4] [1 3 5]]\n\n(d/reduce (d/with (take 5)\n                  (d/drains {:min (d/drain min ##Inf)\n                             :max (d/drain max ##-Inf)}))\n          [3 1 4 1 5 9 2])\n;=\u003e {:min 1, :max 5}\n```\n\nIn particular, `(d/with xf (d/drain op val))` is equivalent to `(d/drain xf op val)`.\n\n`d/with` is so powerful to make existing drains reusable in various ways:\n\n```clj\n(def countries\n  [{:name \"Canada\" :area 9984 :population 36 :continent \"North America\"}\n   {:name \"China\" :area 9634 :population 1390 :continent \"Asia\"}\n   {:name \"Germany\" :area 357 :population 82 :continent \"Europe\"}\n   {:name \"Japan\" :area 377 :population 126 :continent \"Asia\"}\n   {:name \"UK\" :area 244 :population 63 :continent \"Europe\"}\n   {:name \"USA\" :area 9628 :population 327 :continent \"North America\"}])\n\n(def sum (d/drain +))\n\n;; total population\n(d/reduce (d/with (map :population) sum) countries)\n;=\u003e 2024\n\n;; total area\n(d/reduce (d/with (map :area) sum) countries)\n;=\u003e 30224\n\n;; total are in Europe\n(d/reduce (d/with (comp (filter #(= (:continent %) \"Europe\"))\n                        (map :area))\n                  sum)\n          countries)\n;=\u003e 601\n```\n\nNote that which drain you attach a transducer to may cause a different result in some cases. For example:\n\n```clj\n(d/reduce (d/with (take 5)\n                  (d/drains {:evens (d/drain (filter even?) conj [])\n                             :odds (d/drain (filter odd?) conj [])}))\n          (range 20))\n;=\u003e {:evens [0 2 4], :odds [1 3]}\n\n(d/reduce (d/drains {:evens (d/drain (comp (filter even?) (take 5)) conj [])\n                     :odds (d/drain (comp (filter odd?) (take 5)) conj [])})\n          (range 20))\n;=\u003e {:evens [0 2 4 6 8], :odds [1 3 5 7 9]}\n```\n\n```clj\n(d/reduce (d/with (take 5)\n                  (d/group-by #(rem % 3)\n                              (d/drain conj)))\n          (range 20))\n;=\u003e {0 [0 3], 1 [1 4], 2 [2]}\n\n(d/reduce (d/group-by #(rem % 3)\n                      (d/with (take 5)\n                              (d/drain conj)))\n          (range 20))\n;=\u003e {0 [0 3 6 9 12],\n;    1 [1 4 7 10 13],\n;    2 [2 5 8 11 14]}\n```\n\n### `d/reductions`, `d/fold`\n\nThe library also provides some more aggregation fns such as `d/reductions` and `d/fold` besides `d/reduce`. They can be used almost the same as Clojure's counterparts except that they accept a drain instead of a reducing fn:\n\n```clj\n(d/reductions (d/drains {:sum (d/drain +)\n                         :count (d/drain (map (constantly 1)) + 0)})\n              (range 5))\n;=\u003e ({:sum 0, :count 0}\n;    {:sum 0, :count 1}\n;    {:sum 1, :count 2}\n;    {:sum 3, :count 3}\n;    {:sum 6, :count 4}\n;    {:sum 10, :count 5})\n\n(d/fold 2048 + (d/drain +) (vec (range 100000)))\n;=\u003e 4999950000\n```\n\n## Performance\n\nThe Drains runs very efficiently in spite of its expressiveness and flexibility in design.\n\nDrains suppress memory allocation during the aggregation as much as possible and their implementations are well optimized for their typical use cases. So, they are usually almost equal to (or sometimes even better than) Clojure's counterparts in performance:\n\n```clj\n(require '[criterium.core :refer [quick-bench]])\n\n;; Drains\n\n(quick-bench (d/reduce (d/drains [(d/drain min ##Inf) (d/drain max ##-Inf)]) (range 1000000)))\n;; Evaluation count : 24 in 6 samples of 4 calls.\n;;              Execution time mean : 28.793182 ms\n;;     Execution time std-deviation : 950.228568 µs\n;;    Execution time lower quantile : 27.811876 ms ( 2.5%)\n;;    Execution time upper quantile : 29.787913 ms (97.5%)\n;;                    Overhead used : 8.015921 ns\n\n;; corresponding code in clojure.core\n\n(quick-bench (reduce (fn [[mi ma] x] [(min mi x) (max ma x)]) [##Inf ##-Inf] (range 1000000)))\n;; Evaluation count : 18 in 6 samples of 3 calls.\n;;              Execution time mean : 38.313381 ms\n;;     Execution time std-deviation : 1.375499 ms\n;;    Execution time lower quantile : 36.656866 ms ( 2.5%)\n;;    Execution time upper quantile : 40.239558 ms (97.5%)\n;;                    Overhead used : 8.015921 ns\n```\n\n```clj\n;; Drains\n\n(quick-bench (d/reduce (d/group-by #(rem % 10) (d/drain conj)) (range 1000000)))\n;; Evaluation count : 6 in 6 samples of 1 calls.\n;;              Execution time mean : 119.204513 ms\n;;     Execution time std-deviation : 24.679129 ms\n;;    Execution time lower quantile : 89.878243 ms ( 2.5%)\n;;    Execution time upper quantile : 149.656373 ms (97.5%)\n;;                    Overhead used : 8.015921 ns\n\n;; corresponding code in clojure.core\n\n(quick-bench (group-by #(rem % 10) (range 1000000)))\n;; Evaluation count : 6 in 6 samples of 1 calls.\n;;              Execution time mean : 222.926809 ms\n;;     Execution time std-deviation : 31.079101 ms\n;;    Execution time lower quantile : 187.375581 ms ( 2.5%)\n;;    Execution time upper quantile : 257.340992 ms (97.5%)\n;;                    Overhead used : 8.015921 ns\n```\n\n## Related works\n\n- [babbage](https://github.com/ReadyForZero/babbage)\n    - Very much resembles Drains except for its computation graph facilities\n    - Provides a little bit complicated API due to lack of transducers integration (since it was released long before Clojure introduced transducers in 1.7!)\n- [xforms](https://github.com/cgrand/xforms)\n    - Provides various utility transducers and reducing fns\n    - Those utilities can be effectively used from Drains through drain abstraction\n- [parallel](https://github.com/reborg/parallel)\n    - Defines a version of aggregation fns enabled to perform parallel execution a la `clojure.core.reducers/fold`\n    - Although Drains also provides the `fold` fn for that purpose, `parallel` functions often show better performance for parallel execution in paticular.\n\n## License\n\nCopyright © 2018 Shogo Ohta\n\nDistributed under the Eclipse Public License either version 1.0 or (at\nyour option) any later version.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fathos%2Fdrains","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fathos%2Fdrains","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fathos%2Fdrains/lists"}