{"id":33231765,"url":"https://github.com/fosskers/transducers","last_synced_at":"2026-01-26T13:06:52.198Z","repository":{"id":66324958,"uuid":"591545127","full_name":"fosskers/transducers","owner":"fosskers","description":"Transducers: Ergonomic, efficient data processing.","archived":false,"fork":false,"pushed_at":"2026-01-14T23:02:23.000Z","size":372,"stargazers_count":150,"open_issues_count":3,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2026-01-16T23:55:55.797Z","etag":null,"topics":["functional-programming","lisp","transducers"],"latest_commit_sha":null,"homepage":"","language":"Common Lisp","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fosskers.png","metadata":{"files":{"readme":"README.org","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-01-21T03:19:38.000Z","updated_at":"2026-01-14T23:02:02.000Z","dependencies_parsed_at":"2023-12-31T15:28:11.819Z","dependency_job_id":"53230a3b-2896-4ca0-aac0-9477774657b3","html_url":"https://github.com/fosskers/transducers","commit_stats":{"total_commits":190,"total_committers":2,"mean_commits":95.0,"dds":"0.0052631578947368585","last_synced_commit":"da081b71c20c38f9939fda1863022dc5f2ed97d1"},"previous_names":["fosskers/transducers"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/fosskers/transducers","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fosskers%2Ftransducers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fosskers%2Ftransducers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fosskers%2Ftransducers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fosskers%2Ftransducers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fosskers","download_url":"https://codeload.github.com/fosskers/transducers/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fosskers%2Ftransducers/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28778801,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-26T11:46:04.308Z","status":"ssl_error","status_checked_at":"2026-01-26T11:46:02.664Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["functional-programming","lisp","transducers"],"created_at":"2025-11-16T18:00:20.906Z","updated_at":"2026-01-26T13:06:52.176Z","avatar_url":"https://github.com/fosskers.png","language":"Common Lisp","funding_links":[],"categories":["Python ##","Miscellaneous ##"],"sub_categories":[],"readme":"#+title: Transducers: Ergonomic, efficient data processing\n\n#+begin_quote\nI think Transducers are a fundamental primitive that decouples critical logic\nfrom list/sequence processing, and if I had to do Clojure all over I would put\nthem at the bottom.\n\n-- Rich Hickey\n#+end_quote\n\nTransducers are an ergonomic and extremely memory-efficient way to process a\ndata source. Here \"data source\" means simple collections like Lists or Vectors,\nbut also potentially large files or generators of infinite data.\n\nTransducers...\n\n- allow the chaining of operations like ~map~ and ~filter~ without allocating memory between each step.\n- aren't tied to any specific data type; they need only be implemented once.\n- vastly simplify \"data transformation code\".\n- have nothing to do with \"lazy evaluation\".\n- are a joy to use!\n\nExample: /While skipping every second line of a file, sum the lengths of only\nevenly-lengthed lines./\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n\n(transduce\n  ;; How do we want to process each element?\n  (comp (step 2) (map #'length) (filter #'evenp))\n  ;; How do we want to combine all the elements together?\n  #'+\n  ;; What's our original data source?\n  #p\"README.org\")\n#+end_src\n\n#+RESULTS:\n: 10696\n\nLooking for Transducers in other Lisps? Check out the [[https://github.com/fosskers/transducers.el][Emacs Lisp]] and [[https://github.com/fosskers/transducers.fnl][Fennel]] implementations!\n\n* Table of Contents :TOC_5_gh:noexport:\n- [[#compatibility][Compatibility]]\n- [[#history-and-motivation][History and Motivation]]\n- [[#installation][Installation]]\n- [[#usage-and-theory][Usage and Theory]]\n  - [[#importing][Importing]]\n  - [[#transducers-reducers-and-sources][Transducers, Reducers, and Sources]]\n  - [[#processing-json-data][Processing JSON Data]]\n  - [[#fset-immutable-collections][Fset: Immutable Collections]]\n- [[#api][API]]\n  - [[#transducers][Transducers]]\n    - [[#pass-map][pass, map]]\n    - [[#filter-filter-map][filter, filter-map]]\n    - [[#unique-unique-by-dedup][unique, unique-by, dedup]]\n    - [[#drop-drop-while-take-take-while][drop, drop-while, take, take-while]]\n    - [[#uncons-concatenate-flatten][uncons, concatenate, flatten]]\n    - [[#segment-window-group-by][segment, window, group-by]]\n    - [[#intersperse-enumerate-step-scan][intersperse, enumerate, step, scan]]\n    - [[#once][once]]\n    - [[#log][log]]\n    - [[#sexp][sexp]]\n    - [[#from-csv-into-csv][from-csv, into-csv]]\n    - [[#safe][safe]]\n  - [[#reducers][Reducers]]\n    - [[#cons-snoc][cons, snoc]]\n    - [[#vector-bit-vector][vector, bit-vector]]\n    - [[#string-base-string][string, base-string]]\n    - [[#hash-table][hash-table]]\n    - [[#count-quantities-ratio][count, quantities, ratio]]\n    - [[#average-median-variance][average, median, variance]]\n    - [[#any-all][any?, all?]]\n    - [[#partition][partition]]\n    - [[#first-last-find][first, last, find]]\n    - [[#fold][fold]]\n    - [[#for][for]]\n  - [[#sources][Sources]]\n    - [[#ints-random][ints, random]]\n    - [[#cycle-repeat-shuffle][cycle, repeat, shuffle]]\n    - [[#plist][plist]]\n    - [[#reversed][reversed]]\n  - [[#utilities][Utilities]]\n    - [[#comp-const][comp, const]]\n    - [[#reduced-reduced-reduced-val][reduced, reduced?, reduced-val]]\n- [[#example-gallery][Example Gallery]]\n  - [[#reading-lines-from-a-file][Reading lines from a File]]\n  - [[#process-stdout-from-a-shell-call][Process STDOUT from a Shell Call]]\n  - [[#standard-deviation][Standard Deviation]]\n  - [[#shuffle-a-vector][Shuffle a Vector]]\n  - [[#reducing-into-property-lists-and-assocation-lists][Reducing into Property Lists and Assocation Lists]]\n  - [[#json-calculating-average-age][JSON: Calculating average age]]\n- [[#writing-your-own-primitives][Writing your own Primitives]]\n  - [[#transducers-1][Transducers]]\n    - [[#map---a-simple-transformation][map - A simple transformation]]\n    - [[#filter---ignoring-input][filter - Ignoring input]]\n    - [[#take-while---short-circuiting][take-while - Short-circuiting]]\n    - [[#unique---stateful-transduction][unique - Stateful transduction]]\n  - [[#reducers-1][Reducers]]\n    - [[#count---simple-cumulative-state][count - Simple cumulative state]]\n    - [[#cons---some-post-processing][cons - Some post-processing]]\n    - [[#any---short-circuiting][any? - Short-circuiting]]\n  - [[#sources-1][Sources]]\n- [[#limitations][Limitations]]\n- [[#resources][Resources]]\n\n* Compatibility\n\n- Compiles: does =(asdf:load-system :transducers)= succeed?\n- Tests: does =(asdf:test-system :transducers)= succeed?\n\n| Compiler  | Compiles? | Tests? | Notes                      |\n|-----------+-----------+--------+----------------------------|\n| SBCL      | ✅        | ✅     |                            |\n| ECL       | ✅        | ✅     |                            |\n| Clasp     | ✅        | -      |                            |\n| ABCL      | ✅        | ❌     | [[https://github.com/armedbear/abcl/issues/675][No TCO]] in recursive =labels= |\n| CCL       | ✅        | ✅     |                            |\n| [[https://gitlab.com/gnu-clisp/clisp][Clisp]]     | ✅        | -      | Supports TCO but not [[https://gitlab.com/gnu-clisp/clisp/-/merge_requests/3][PLN]]   |\n| Allegro   | ✅        | ✅     |                            |\n| LispWorks | ✅        | -      |                            |\n\n[[https://wiki.archlinux.org/title/Common_Lisp#Historical][Historical implementations]] are not considered.\n\n* History and Motivation\n\nOriginally invented in Clojure and adapted to Scheme as SRFI-171, Transducers\nare an excellent way to think about - and efficiently operate on - collections\nor streams of data. Transduction operations are strict and don't involve\n\"laziness\" or \"thunking\" in any way, yet only process the exact amount of data\nyou ask them to.\n\nThis library draws inspiration from both the original Clojure and SRFI-171,\nwhile adding many other convenient operations commonly found in other languages.\n\n* Installation\n\nThis library is available through [[https://github.com/fosskers/vend][vend]], as well as [[https://quickdocs.org/cl-transducers][Quicklisp]] and [[https://ultralisp.org/projects/fosskers/cl-transducers][Ultralisp]].\n\n* Usage and Theory\n\n** Importing\n\nSince this library reuses some symbol names also found in =:cl=, it is expected\nthat you import =transducers= as follows in your =defpackage=:\n\n#+begin_src lisp\n(defpackage foo\n  (:use :cl)\n  (:local-nicknames (:t :transducers)))\n#+end_src\n\nYou can then make relatively clean calls like:\n\n#+begin_src lisp\n(t:transduce (t:map #'1+) #'t:vector '(1 2 3))\n;; =\u003e #(2 3 4)\n#+end_src\n\nHowever, many of the examples below use ~(in-package :transducers)~ for brevity in\nthe actual function calls. You should still use a nickname in your own code.\n\n** Transducers, Reducers, and Sources\n\n#+begin_src lisp\n;; The fundamental pattern.\n(transduce \u003ctransducer-chain\u003e \u003creducer\u003e \u003csource\u003e)\n#+end_src\n\nData processing largely has three concerns:\n\n1. Where is my data coming from? (sources)\n2. What do I want to do to each element? (transducers)\n3. How do I want to collect the results? (reducers)\n\nEach full \"transduction\" requires all three. We pass one of each to the\n=transduce= function, which drives the process. It knows how to pull values from\nthe source, feed them through the transducer chain, and wrap everything together\nvia the reducer.\n\n- Typical transducers are =map=, =filter=, and =take=.\n- Typical reducers are =+=, =count=, =t:cons=, and =fold=.\n- Typical sources are lists, vectors, strings, hash tables, and files.\n\n/Generators/ are a special kind of source that yield infinite data. Typical\ngenerators are =repeat=, =cycle=, and =ints=.\n\nLet's sum the squares of the first 1000 odd integers:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n\n(transduce\n (comp (filter #'oddp)             ;; (2) Keep only odd numbers.\n       (take 1000)                 ;; (3) Keep the first 1000 filtered odds.\n       (map (lambda (n) (* n n)))) ;; (4) Square those 1000.\n #'+       ;; (5) Reducer: Add up all the squares.\n (ints 1)) ;; (1) Source: Generate all positive integers.\n#+end_src\n\n#+RESULTS:\n: 1333333000\n\nTwo things of note here:\n\n1. =comp= is used here to chain together different transducer steps. Notice that\n   the order appears \"backwards\" from usual function composition. It may help to\n   imagine that =comp= is acting like the =-\u003e\u003e= macro here. =comp= is supplied here as\n   a convenience; you're free to use =alexandria:compose= if you wish.\n2. The reduction via =+= is listed as Step 5, but really it's occuring throughout\n   the transduction process. Each value that makes it through the composed\n   transducer chain is immediately added to an internal accumulator.\n\nExplore the other transducers and reducers to see what's possible! You'll never\nwrite a =loop= again.\n\n** Processing JSON Data\n\nThe system =transducers/jzon= provides automatic JSON streaming support via the\n[[https://github.com/Zulu-Inuoe/jzon][jzon]] library. Like =transducers= itself, it is expected that you import this\nsystem with a nickname:\n\n#+begin_src lisp\n(:local-nicknames (#:j #:transducers/jzon))\n#+end_src\n\nOnly two functions are exposed: =read= and =write=.\n\n- =read= is a /source/ that accepts a pathname, open stream, or a string. It\n  produces parsed JSON values as Lisp types. JSON Objects become Hash Tables.\n- =write= is a /reducer/ that expects an open stream. It writes the stream of Lisp\n  types into their logical JSON equivalents.\n\nHere is a simple example of reading some JSON data from a string, doing nothing\nto it, and outputting it again to a new string:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n\n(with-output-to-string (stream)\n  (transduce #'pass\n             (transducers/jzon:write stream)\n             (transducers/jzon:read \"[{\\\"name\\\": \\\"A\\\"}, {\\\"name\\\": \\\"B\\\"}]\")))\n#+end_src\n\n#+RESULTS:\n: [{\"name\":\"A\"},{\"name\":\"B\"}]\n\nNote that the JSON data _must_ be a JSON array. There is otherwise no size limit;\nthe library can handle any amount of JSON input.\n\nFor more examples, see the Gallery below.\n\n** Fset: Immutable Collections\n\nThe system =transducers/fset= provides support for the [[https://gitlab.common-lisp.net/fset/fset][Fset library]] of immutable\ncollections. It's expected that you import this system with a nickname:\n\n#+begin_src lisp\n(:local-nicknames (#:s #:transducers/fset))\n#+end_src\n\nReducers are provided for each of its main types: ~set~, ~map~, ~seq~, and ~bag~.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n\n(transduce (map #'1+) #'transducers/fset:set (fset:set 1 2 3 1))\n#+end_src\n\n#+RESULTS:\n: #{ 2 3 4 }\n\n* API\n\nThe examples here use ~(in-package :transducers)~ for brevity in the actual\nfunction calls and to allow them to be runnable directly in this README, but as\nmentioned above it's recommended to nickname the library to ~:t~ due to some\noverlap with ~:cl~.\n\n** Transducers\n\nTransducers describe how to alter the items of some stream of values. Some\ntransducers, like ~take~, can short-circuit.\n\nMultiple transducer functions can be chained together with ~comp~.\n\n*** pass, map\n\n=pass=: Just pass along each value of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass #'cons '(1 2 3))\n#+end_src\n\n#+RESULTS:\n: (1 2 3)\n\n=map=: Apply a function F to all elements of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (map #'1+) #'cons '(1 2 3))\n#+end_src\n\n#+RESULTS:\n: (2 3 4)\n\n*** filter, filter-map\n\n=filter=: Only keep elements from the transduction that satisfy PRED.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (filter #'evenp) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: (2 4)\n\n=filter-map=: Apply a function F to the elements of the transduction, but only\nkeep results that are non-nil.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (filter-map #'cl:first) #'cons '(() (2 3) () (5 6) () (8 9)))\n#+end_src\n\n#+RESULTS:\n: (2 5 8)\n\n*** unique, unique-by, dedup\n\n=unique=: Only allow values to pass through the transduction once each. Stateful;\nthis uses a set internally so could get quite heavy if you're not careful.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'unique #'cons '(1 2 1 3 2 1 2 \"abc\"))\n#+end_src\n\n#+RESULTS:\n: (1 2 3 \"abc\")\n\n=unique-by=: Only allow values to pass through the transduction once each,\ndetermined by some key-mapping function. The function is only used to map the\nvalues to something they should be compared to; the original values themselves\nare what is passed through. Stateful; this uses a Hash Table internally so could\nget quite heavy if you're not careful.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (unique-by #'cdr) #'cons '((:a . 1) (:b . 2) (:c . 1) (:d . 3)))\n#+end_src\n\n#+RESULTS:\n: ((:A . 1) (:B . 2) (:D . 3))\n\n=dedup=: Remove adjacent duplicates from the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'dedup #'cons '(1 1 1 2 2 2 3 3 3 4 3 3))\n#+end_src\n\n#+RESULTS:\n: (1 2 3 4 3)\n\n*** drop, drop-while, take, take-while\n\n=drop=: Drop the first N elements of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (drop 3) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: (4 5)\n\n=drop-while=: Drop elements from the front of the transduction that satisfy PRED.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (drop-while #'evenp) #'cons '(2 4 6 7 8 9))\n#+end_src\n\n#+RESULTS:\n: (7 8 9)\n\n=take=: Keep only the first N elements of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 3) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: (1 2 3)\n\n=take-while=: Keep only elements which satisfy a given PRED, and stop the\ntransduction as soon as any element fails the test.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take-while #'evenp) #'cons '(2 4 6 8 9 2))\n#+end_src\n\n#+RESULTS:\n: (2 4 6 8)\n\n*** uncons, concatenate, flatten\n\n=uncons=: Split up a transduction of cons cells.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'uncons #'cons '((:a . 1) (:b . 2) (:c . 3)))\n#+end_src\n\n#+RESULTS:\n: (:A 1 :B 2 :C 3)\n\n=concatenate=: Concatenate all the sublists, subvectors, or substrings in the\ntransduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'concatenate #'cons '((1 2 3) (4 5 6) (7 8 9)))\n#+end_src\n\n#+RESULTS:\n: (1 2 3 4 5 6 7 8 9)\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp #'concatenate (intersperse #\\!))\n           #'string '(\"hello\" \"there\"))\n#+end_src\n\n#+RESULTS:\n: h!e!l!l!o!t!h!e!r!e\n\n=flatten=: Entirely flatten all lists in the transduction, regardless of nesting.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'flatten #'cons '((1 2 3) 0 (4 (5) 6) 0 (7 8 9) 0))\n#+end_src\n\n#+RESULTS:\n: (1 2 3 0 4 5 6 0 7 8 9 0)\n\n*** segment, window, group-by\n\n=segment=: Partition the input into lists of N items. If the input stops, flush\nany accumulated state, which may be shorter than N.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (segment 3) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: ((1 2 3) (4 5))\n\n=window=: Yield N-length windows of overlapping values. This is different from\n~segment~ which yields non-overlapping windows. If there were fewer items in the\ninput than N, then this yields nothing.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (window 3) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: ((1 2 3) (2 3 4) (3 4 5))\n\n=group-by=: Group the input stream into sublists via some function F. The cutoff\ncriterion is whether the return value of F changes between two consecutive\nelements of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (group-by #'evenp) #'cons '(2 4 6 7 9 1 2 4 6 3))\n#+end_src\n\n#+RESULTS:\n: ((2 4 6) (7 9 1) (2 4 6) (3))\n\n*** intersperse, enumerate, step, scan\n\n=intersperse=: Insert an ELEM between each value of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (intersperse 0) #'cons '(1 2 3))\n#+end_src\n\n#+RESULTS:\n: (1 0 2 0 3)\n\n=enumrate=: Index every value passed through the transduction into a cons pair.\nStarts at 0.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'enumerate #'cons '(\"a\" \"b\" \"c\"))\n#+end_src\n\n#+RESULTS:\n: ((0 . \"a\") (1 . \"b\") (2 . \"c\"))\n\n=step=: Only yield every Nth element of the transduction. The first element of the\ntransduction is always included.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (step 2) #'cons '(1 2 3 4 5 6 7 8 9))\n#+end_src\n\n#+RESULTS:\n: (1 3 5 7 9)\n\n=scan=: Build up successsive values from the results of previous applications of a\ngiven function F.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (scan #'+ 0) #'cons '(1 2 3 4))\n#+end_src\n\n#+RESULTS:\n: (0 1 3 6 10)\n\n*** once\n\n=once=: Inject some ITEM onto the front of the transduction.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp (filter (lambda (n) (\u003e n 10)))\n                 (once 'hello)\n                 (take 3))\n           #'cons (ints 1))\n#+end_src\n\n#+RESULTS:\n: (HELLO 11 12)\n\n*** log\n\n=log=: Call some LOGGER function for each step of the transduction. The LOGGER\nmust accept the running results and the current element as input. The original\nitems of the transduction are passed through as-is.\n\n#+begin_src lisp :results output :exports both\n(in-package :transducers)\n(transduce (log (lambda (_ n) (format t \"Got: ~a~%\" n))) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: Got: 1\n: Got: 2\n: Got: 3\n: Got: 4\n: Got: 5\n\nThese are STDOUT results. The actual return value is the result of the reducer,\nin this case ~cons~, thus a list.\n\n*** sexp\n\n=sexp=: Interpret the data stream as S-expressions, yielding one at a time. Can\neither process a stream of individual characters directly from a string...\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'sexp #'cons \"(+ 1 1) (+ 2 2) (+ 3 (* 4 5))\")\n#+end_src\n\n#+RESULTS:\n: (\"(+ 1 1)\" \"(+ 2 2)\" \"(+ 3 (* 4 5))\")\n\n...or of chunks of strings, for instance from a file stream.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'sexp #'cons '(\"(+ 1\" \" 1) (+ 2 2)\"))\n#+end_src\n\n#+RESULTS:\n: (\"(+ 1 1)\" \"(+ 2 2)\")\n\n*** from-csv, into-csv\n\n=from-csv=: Interpret the data stream as CSV data.\n\nThe first item found is assumed to be the header list, and it will be used to\nconstruct useable hashtables for all subsequent items.\n\nNote: This function makes no attempt to convert types from the original parsed\nstrings. If you want numbers, you will need to further parse them yourself.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp #'from-csv\n                 (map (lambda (hm) (gethash \"Name\" hm))))\n           #'cons '(\"Name,Age\" \"Alice,35\" \"Bob,26\"))\n#+end_src\n\n#+RESULTS:\n: (\"Alice\" \"Bob\")\n\n=into-csv=: Given a sequence of HEADERS, rerender each item in the data stream\ninto a CSV string. It's assumed that each item in the transduction is a hash\ntable whose keys are strings that match the values found in HEADERS.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp #'from-csv\n                 (into-csv '(\"Name\" \"Age\")))\n           #'cons '(\"Name,Age,Hair\" \"Alice,35,Blond\" \"Bob,26,Black\"))\n#+end_src\n\n#+RESULTS:\n: (\"Name,Age\" \"Alice,35\" \"Bob,26\")\n\n*** safe\n\n=safe:= Given a transducer that may error, wrap it in =restart-case= to provide\nvarious ways of skipping or handling that error.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(handler-bind ((error #'(lambda (c) (invoke-restart 'next-item))))\n  (transduce (safe (map (lambda (item) (if (= item 1) (error \"Mercy!\") item))))\n             #'cons '(0 1 2 3)))\n#+end_src\n\n#+RESULTS:\n: (0 2 3)\n\n** Reducers\n\nReducers describe how to fold the stream of items down into a single result, be\nit either a new collection or a scalar.\n\nSome reducers, like ~first~, can also force the entire transduction to\nshort-circuit.\n\n*** cons, snoc\n\n=cons=: Collect all results as a list.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass #'cons '(1 2 3))\n#+end_src\n\n#+RESULTS:\n: (1 2 3)\n\n=snoc=: Collect all results as a list, but results are reversed. In theory,\nslightly more performant than ~cons~ since it performs no final reversal.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass #'snoc '(1 2 3))\n#+end_src\n\n#+RESULTS:\n: (3 2 1)\n\n*** vector, bit-vector\n\n=vector=: Collect a stream of values into a vector.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass #'vector '(1 2 3))\n#+end_src\n\n#+RESULTS:\n: #(1 2 3)\n\n=bit-vector=: If the elements are all either 0 or 1, you can reduce into a\n=bit-vector= instead:\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass #'bit-vector '(0 1 0 1))\n#+end_src\n\n#+RESULTS:\n: #*0101\n\n*** string, base-string\n\n=string=: Collect a stream of characters into a single string. The result is the\nideal =(simple-array character (*))= type, typically the most performant form of a\nstring.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (map #'char-upcase) #'string \"I see 富士山.\")\n#+end_src\n\n#+RESULTS:\n: I SEE 富士山.\n\n=base-string=: However, if you know your source characters are all ASCII (rather,\nthe =base-char= range for your particular compiler), then =base-string= often has\nmuch lower memory consumption:\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (map #'char-upcase) #'base-string \"I see Mt. Fuji.\")\n#+end_src\n\n#+RESULTS:\n: I SEE MT. FUJI.\n\n*** hash-table\n\n=hash-table=: Collect a stream of key-value cons pairs into a hash table.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'enumerate #'hash-table '(\"a\" \"b\" \"c\"))\n#+end_src\n\n#+RESULTS:\n: #\u003cCOMMON-LISP:HASH-TABLE :TEST EQUAL :COUNT 3 {1004E83BF3}\u003e\n\n*** count, quantities, ratio\n\n=count=: Count the number of elements that made it through the transduction.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass #'count '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: 5\n\n=quantities=: Count the occurrences of every item in the transduction, given some\nequality predicate.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass (quantities #'eql) '(1 1 1 0 2 4 1 4 9))\n#+end_src\n\n#+RESULTS:\n: #\u003cCOMMON-LISP:HASH-TABLE :TEST EQL :COUNT 5 {10094FE823}\u003e\n\n=ratio=: The percentage of items that satisfied a predicate. The final value will\nalways be between 0 and 1.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass (ratio #'evenp) #(1 2 3 4 5 6))\n#+end_src\n\n#+RESULTS:\n: 1/2\n\n*** average, median, variance\n\n=average=: Calculate the average value of all numeric elements in a transduction.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass #'average '(1 2 3 4 5 6))\n#+end_src\n\n#+RESULTS:\n: 7/2\n\n=median=: Calculate the median value of all elements in a transduction, provided\nthat they are numbers, strings, or characters. The elements are sorted once\nbefore the median is extracted.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass #'median '(1 1 1 0 2 4 1 4 9))\n#+end_src\n\n#+RESULTS:\n: 1\n\n=variance=: Calculate the variance of a stream of values, given their mean.\n\nUsing this to calculate Standard Deviation:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(let* ((nums '(1 2 3 4 5 6 7 8 9 10))\n       (mean (transduce #'pass #'average nums)))\n  (sqrt (transduce #'pass (variance mean) nums)))\n#+end_src\n\n#+RESULTS:\n: 2.8722813\n\n*** any?, all?\n\n=any?=: Yield =t= if any element in the transduction satisfies PRED. Short-circuits\nthe transduction as soon as the condition is met.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass (any? #'evenp) '(1 3 5 7 9 2))\n#+end_src\n\n#+RESULTS:\n: T\n\n=all?=: Yield =t= if all elements of the transduction satisfy PRED. Short-circuits\nwith NIL if any element fails the test.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass (all? #'oddp) '(1 3 5 7 9))\n#+end_src\n\n#+RESULTS:\n: T\n\n*** partition\n\n=partition=: Given some predicate function, split the stream and collect its\nvalues into two lists, those items that passed the check, and those that failed.\nYou must use =multiple-value-bind= or similar to receive both lists.\n\n#+begin_src lisp :exports both :results verbatim\n(in-package :transducers)\n(transduce #'pass (partition #'evenp) '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: (2 4), (1 3 5)\n\n*** first, last, find\n\n=first=: Yield the first value of the transduction. As soon as this first value is\nyielded, the entire transduction stops.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce (filter #'oddp) #'first '(2 4 6 7 10))\n#+end_src\n\n#+RESULTS:\n: 7\n\n=last=: Yield the last value of the transduction.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass #'last '(2 4 6 7 10))\n#+end_src\n\n#+RESULTS:\n: 10\n\n=find=: Find the first element in the transduction that satisfies a given PRED.\nYields NIL if no such element were found.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass (find #'evenp) '(1 3 5 6 9))\n#+end_src\n\n#+RESULTS:\n: 6\n\n*** fold\n\n=fold=: The fundamental reducer. ~fold~ creates an ad-hoc reducer based on a given\n2-argument function. An optional SEED value can also be given as the initial\naccumulator value, which also becomes the return value in case there were no\ninput left in the transduction.\n\nFunctions like ~+~ and ~*~ are automatically valid reducers, because they yield sane\nvalues even when given 0 or 1 arguments. Other functions like ~cl:max~ cannot be\nused as-is as reducers since they can't be called without arguments. For\nfunctions like this, ~fold~ is appropriate.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass (fold #'cl:max) '(1 2 3 4 1000 5 6))\n#+end_src\n\n#+RESULTS:\n: 1000\n\nWith a seed:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass (fold #'cl:max 0) '())\n#+end_src\n\n#+RESULTS:\n: 0\n\nIn Clojure this function is called =completing=.\n\n*** for\n\n=for=: Run through every item in a transduction for their side effects. Throws\naway all results and yields a final =t=.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (map #'1+) (for (lambda (n) (format t \"~a~%\" n))) #(1 2 3 4))\n#+end_src\n\n#+RESULTS:\n: T\n\n** Sources\n\nData is pulled in an on-demand fashion from /Sources/. They can be either finite\nor infinite in length. A list is an example of a simple Source, but you can also\npull from files and endless number generators.\n\n*** ints, random\n\n=ints=: Yield all integers, beginning with START and advancing by an optional STEP\nvalue which can be positive or negative. If you only want a specific range\nwithin the transduction, then use ~take-while~ within your transducer chain.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 10) #'cons (ints 0 :step 2))\n#+end_src\n\n#+RESULTS:\n: (0 2 4 6 8 10 12 14 16 18)\n\n=random=: Yield an endless stream of random numbers, based on a given LIMIT.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 20) #'cons (random 10))\n#+end_src\n\n#+RESULTS:\n: (8 0 5 6 6 2 2 4 2 7 9 2 0 0 2 4 4 9 9 9)\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 5) #'cons (random 1.0))\n#+end_src\n\n#+RESULTS:\n: (0.4115485 0.35940528 0.0056368113 0.31019592 0.4214077)\n\n*** cycle, repeat, shuffle\n\n=cycle=: Yield the values of a given SEQ endlessly.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 10) #'cons (cycle '(1 2 3)))\n#+end_src\n\n#+RESULTS:\n: (1 2 3 1 2 3 1 2 3 1)\n\n=repeat=: Endlessly yield a given ITEM.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 4) #'cons (repeat 9))\n#+end_src\n\n#+RESULTS:\n: (9 9 9 9)\n\n=shuffle=: Endlessly yield random elements from a given vector.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 5) #'cons (shuffle #(\"Alice\" \"Bob\" \"Dennis\")))\n#+end_src\n\n#+RESULTS:\n: (\"Alice\" \"Bob\" \"Alice\" \"Dennis\" \"Bob\")\n\nRecall also that strings are vectors too:\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (take 15) #'string (shuffle \"Númenor\"))\n#+end_src\n\n#+RESULTS:\n: eeúúrúmnnremmno\n\n*** plist\n\n=plist=: Yield key-value pairs from a Property List, usually known as a 'plist'.\nThe pairs are passed as a cons cell.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce (map #'cdr) #'+ (plist '(:a 1 :b 2 :c 3)))\n#+end_src\n\n#+RESULTS:\n: 6\n\nSee also the ~uncons~ transducer for another way to handle incoming cons cells.\n\n*** reversed\n\n=reversed=: Yield a vector's elements in reverse order.\n\n#+begin_src lisp :exports both :results verbatim\n(in-package :transducers)\n(transduce (take 2) #'cons (reversed #(1 2 3 4)))\n#+end_src\n\n#+RESULTS:\n: (4 3)\n\nRecall that strings are also vectors.\n\n#+begin_src lisp :exports both :results verbatim\n(in-package :transducers)\n(transduce #'pass #'string (reversed \"Hello\"))\n#+end_src\n\n#+RESULTS:\n: olleH\n\n** Utilities\n\n*** comp, const\n\n=comp=: Function composition as a macro. You can pass as many functions as you\nlike and they are applied from right to left.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(funcall (comp #'length #'reverse) #(1 2 3))\n#+end_src\n\n#+RESULTS:\n: 3\n\nFor transducer functions specifically, they are /composed/ from right to left, but\ntheir effects are /applied/ from left to right. This is due to how the reducer\nfunction is chained through them all internally via ~transduce~.\n\nNotice here how ~drop~ is clearly applied first:\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp (drop 3) (take 2)) #'cons '(1 2 3 4 5 6))\n#+end_src\n\n#+RESULTS:\n: (4 5)\n\n=const=: Return a function that ignores its argument and returns ITEM instead.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(funcall (comp (const 108) (lambda (n) (* 2 n)) #'1+) 1)\n#+end_src\n\n#+RESULTS:\n: 108\n\n*** reduced, reduced?, reduced-val\n\nWhen writing your own transducers and reducers, these functions allow you to\nshort-circuit the entire operation.\n\nHere is a simplified definition of ~first~:\n\n#+begin_src lisp :exports code\n(in-package :transducers)\n(defun first (\u0026optional (acc nil a-p) (input nil i-p))\n  (cond ((and a-p i-p) (reduced input))\n        ((and a-p (not i-p)) acc)\n        (t acc)))\n#+end_src\n\nYou can see ~reduced~ being used to wrap the return value. ~transduce~ sees this\nwrapping and immediately halts further processing.\n\n~reduced?~ and ~reduced-val~ can similarly be used (mostly within transducer\nfunctions) to check if some lower transducer (or the reducer) has signaled a\nshort-circuit, and if so potentially perform some clean-up. This is important\nfor transducers that carry internal state.\n\n* Example Gallery\n\n** Reading lines from a File\n\nPathnames can be passed as-is as a Source. This yields their lines one by one.\n\nCounting words:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce (comp (map #'str:words)\n                 #'concatenate)\n           #'count #p\"README.org\")\n#+end_src\n\n#+RESULTS:\n: 3661\n\n** Process STDOUT from a Shell Call\n\n#+begin_src lisp :exports both :results output\n(in-package :transducers)\n(with-open-stream (stream (sb-ext:process-output\n                           (sb-ext:run-program \"/usr/bin/fd\" '(\"lisp\")\n                                               :output :stream\n                                               :wait nil)))\n  (transduce (map #'string-upcase) (for #'write-line) stream))\n#+end_src\n\n#+RESULTS:\n#+begin_example\nFSET/FSET.LISP\nJZON/JZON.LISP\nRUN-TESTS.LISP\nTESTS/TESTS.LISP\nTRANSDUCERS/BENCHMARKS.LISP\nTRANSDUCERS/CONDITIONS.LISP\nTRANSDUCERS/DEPRECATED.LISP\nTRANSDUCERS/ENTRY.LISP\nTRANSDUCERS/PACKAGE.LISP\nTRANSDUCERS/REDUCERS.LISP\nTRANSDUCERS/SOURCES.LISP\nTRANSDUCERS/TRANSDUCERS.LISP\nTRANSDUCERS/UTILS.LISP\n#+end_example\n\n** Standard Deviation\n\nStandard Deviation is the square root of the Variance, therefore:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(let* ((nums '(1 2 3 4 5 6 7 8 9 10))\n       (mean (transduce #'pass #'average nums)))\n  (sqrt (transduce #'pass (variance mean) nums)))\n#+end_src\n\n#+RESULTS:\n: 2.8722813\n\n** Shuffle a Vector\n\nCombine ~unique~, ~take~, and ~shuffle~ for the generalised effect of shuffling a\nvector.\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp #'unique (take 6))\n           #'vector\n           (shuffle #(\"Alice\" \"Bob\" \"Charles\" \"David\" \"Ellen\" \"Frank\")))\n#+end_src\n\n#+RESULTS:\n: #(\"David\" \"Bob\" \"Charles\" \"Ellen\" \"Frank\" \"Alice\")\n\n** Reducing into Property Lists and Assocation Lists\n\nThere is no special reducer function for plists, because none is needed. If you\nhave a stream of cons cells, you can break it up with ~uncons~ and then collect\nwith ~cons~ as usual:\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce (comp (map (lambda (pair) (cl:cons (car pair) (1+ (cdr pair)))))\n                 #'uncons)\n           #'cons (plist '(:a 1 :b 2 :c 3)))\n#+end_src\n\n#+RESULTS:\n: (:A 2 :B 3 :C 4)\n\nLikewise, Association Lists are already lists-of-cons-cells, so no special\ntreatment is needed:\n\n#+begin_src lisp :results verbatim :exports both\n(in-package :transducers)\n(transduce #'pass #'cons '((:a . 1) (:b . 2) (:c . 3)))\n#+end_src\n\n#+RESULTS:\n: ((:A . 1) (:B . 2) (:C . 3))\n\n** JSON: Calculating average age\n\nSince JSON Objects are parsed as Hash Tables, we use the usual functions to\nretrieve fields we want.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce (filter-map (lambda (ht) (gethash \"age\" ht)))\n           #'average\n           (transducers/jzon:read \"[{\\\"age\\\": 34}, {\\\"age\\\": 25}]\"))\n#+end_src\n\n#+RESULTS:\n: 59/2\n\n* Writing your own Primitives\n\nOne of the advantages of the Transducers pattern is that there is no \"magic\".\nAs you'll see below, it's all just function composition.\n\n** Transducers\n\nA Transducer is a function that _continues the stream_. It operates on one\nelement at a time. It receives input, optionally does something to it, and then\noptionally continues by calling the next function in the chain, or it ignores\nthe current input, or it short-circuits the stream entirely. We'll see examples\nof all of these below.\n\n*** map - A simple transformation\n\nHere is how =map= is implemented in the library. Let's study it to learn the\noverall structure of transducers in general.\n\n#+begin_src lisp\n(defun map (f)  ;; (1) Top-level arguments needed throughout.\n  (lambda (reducer)  ;; (2) The rest of the composed function chain.\n    (lambda (result \u0026optional (input nil i-p))  ;; (3) The main body of the transducer.\n      (if i-p\n          (funcall reducer result (funcall f input))  ;; (4) The primary logic and a call to the next stage.\n          (funcall reducer result)))))  ;; (5) The finalisation pass.\n#+end_src\n\nRecall that =map= would be called like:\n\n#+begin_src lisp :exports both :results verbatim\n(in-package :transducers)\n(transduce (map #'1+) #'cons '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: (2 3 4 5 6)\n\nSo we can see at (1) that the =f= corresponds to the function we're passing in,\nwhich we expect to be applied to all elements of the stream.\n\n(2) might be a surprise. What is =reducer= and where does it come from? Is it the\n=cons= call seen above? Well, it's actually the transducer chain (possibly\ncombined via =comp=), followed by the reducer. Like this:\n\n[[file:transducers.png]]\n\nIt is the call to =transduce= that puts this all together for you.\n\n(3) is what actually gets called during the iteration. The ~\u0026optional (input nil\ni-p)~ may be new to you; this is how Common Lisp handles the potential lack of\noptional arguments.\n\n#+begin_example\n(input nil i-p)\n ^     ^   ^---- Was the optional argument actually given? nil or non-nil.\n |     `---- The default value if the optional arg was missing. Can be anything.\n `---- The name of the optional arg. Either what the user passed, or the default.\n#+end_example\n\nUnfortunately due to \"nil punning\", testing the =input= for =nil= is not enough to\ndetermine if the argument was given or not, since they may have legitimately\npassed =nil=. Hence we need a second signal, named =i-p= here, to do that test.\nClojure and Scheme can pattern match on the number of arguments directly, but\nCommon Lisp cannot. If the =i-p= test fails, then we know the transduction is over.\n\n(4) should be clear; apply =f= and then call the =reducer= the continue the chain.\n\n(5) will become clearer once we've learned about the structure of Reducers. For\nnow, just know that this is the last thing that the top-level =transduce= call\nattempts as it is finalising the result.\n\n*** filter - Ignoring input\n\nWith =map= fresh in your mind, now stare at this:\n\n#+begin_src lisp\n(defun filter (pred)\n  (lambda (reducer)\n    (lambda (result \u0026optional (input nil i-p))\n      (if i-p\n          ;; vvv (4) vvv\n          (if (funcall pred input)\n              (funcall reducer result input) ;; (4a)\n              result) ;; (4b)\n          ;; ^^^ (4) ^^^\n          (funcall reducer result)))))\n#+end_src\n\nPoint (4) in the previous example was the \"meat\", the actual logic of the\ntransducer. Here we see it expanded a bit. Notice that we only continue the\nchain at (4a) if the predicate passed. Otherwise, we _yield the result we were\ngiven_ and directly return, going no further for this particular input element.\nThen, =transduce= will supply the next one. The effect is what we'd expect of\n=filter=; some elements make it through the stream and some don't.\n\n*** take-while - Short-circuiting\n\nSimilar to =filter= is =take-while=, except that the latter halts the stream\nentirely as soon as an element fails the predicate.\n\n#+begin_src lisp\n(defun take-while (pred)\n  (lambda (reducer)\n    (lambda (result \u0026optional (input nil i-p))\n      (if i-p\n          ;; vvv (4) vvv\n          (if (not (funcall pred input))\n              (reduced result)\n              (funcall reducer result input))\n          ;; ^^^ (4) ^^^\n          (funcall reducer result)))))\n#+end_src\n\nHere =reduced= makes its debut. This wraps the given value in a special type that\nsignals to =transduce= that the transduction has been short-circuited and must\nend. Nothing further will be pulled from the Source.\n\n*** unique - Stateful transduction\n\nDespite just being a group of composed functions, individual transducers can\nhold state. Consider =unique=, which is called like:\n\n#+begin_src lisp :exports both :results verbatim\n(in-package :transducers)\n(transduce #'unique #'cons '(1 2 1 3 2 1 2 \"abc\"))\n#+end_src\n\n#+RESULTS:\n: (1 2 3 \"abc\")\n\nHere's its definition:\n\n#+begin_src lisp\n(defun unique (reducer)\n  (let ((seen (make-hash-table :test #'equal)))\n    (lambda (result \u0026optional (input nil i-p)) ;; (3)\n      (if i-p\n          ;; vvv (4) vvv\n          (if (gethash input seen)\n              result\n              (progn (setf (gethash input seen) t)\n                     (funcall reducer result input)))\n          ;; ^^^ (4) ^^^\n          (funcall reducer result)))))\n#+end_src\n\nThere are two immediate differences here:\n\n1. Since =unique= requires no top-level argument (like =map= or =filter=), it is\n   passed directly to =transduce= as =#'unique=. This means we don't need another\n   inner =lambda= and can accept the =reducer= directly.\n2. We can open a =let= before Point (3), and the =seen= Hash Table is then captured\n   by the =lambda=. This has the effect of persisting it between every call of the\n   =lambda= on each element.\n\nOnce again we notice a bare =result= being returned if we've seen the current\nelement already.\n\n** Reducers\n\nA Reducer is a function that _consumes a stream_. It accepts two, one, or no\narguments.\n\n*** count - Simple cumulative state\n\nAn example of =count= being called:\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass #'count '(1 2 3 4 5))\n#+end_src\n\n#+RESULTS:\n: 5\n\nHere is its definition:\n\n#+begin_src lisp\n(defun count (\u0026optional (acc 0 a-p) (input nil i-p))\n  (cond ((and a-p i-p) (1+ acc))   ;; (I) Iterative case. The stream is still running.\n        ((and a-p (not i-p)) acc)  ;; (D) We're done! Do any post-processing here.\n        (t 0)))  ;; (M) \"Monoidal\" / base case.\n#+end_src\n\nSimilar to the Transducer functions, we use the =\u0026optional= trick to test how many\narguments we were given. Let's start from the bottom with the (M) base case.\n=transduce= calls this internally in order to generate an initial value. This\ncorresponds to the =result= seen in the Transducer examples. Since each Reducer\nbehaves differently and we are not using a static type system, we must define\nthe Reducer's unique \"zero value\" here.\n\n(D) is what was hinted at before - this case is called last by =transduce= in\norder to allow the Reducer to do any post-processing before the final value is\nyielded to the user. This is necessary as occasionally the =acc= value grown by\nthe Reducer can be a complicated structure and we may want to sort it, unwrap\nit, etc.\n\n(I) is the usual case and corresponds to some Transducer calling down into the\nReducer with the cumulative state thusfar and the current stream element. The\nReducer then decides what to do with them. In the case of =count=, the element\nitself is ignored and we just add 1 to our growing =acc=.\n\n*** cons - Some post-processing\n\n#+begin_src lisp\n(defun cons (\u0026optional (acc nil a-p) (input nil i-p))\n  (cond ((and a-p i-p) (cl:cons input acc))   ;; (I)\n        ((and a-p (not i-p)) (nreverse acc))  ;; (D)\n        (t '()))) ;; (M)\n#+end_src\n\nHere in (I) we see the =input= actually being saved. This then loops back around\nwithin =transduce=, which pulls the next value from the Source and calls the\nTransducer chain again.\n\nIn (D) we see some realistic post-processing. Since (I) was naively consing, the\norder of our elements is backwards from what we intend. Thus they must be\nreversed once before being yielded to the user.\n\nIn (M) our \"zero value\" is the empty list. Otherwise, what would we be consing\nonto on the first pass of (I)?\n\n*** any? - Short-circuiting\n\n=any?= stops as soon as anything satisfies its predicate.\n\n#+begin_src lisp :exports both\n(in-package :transducers)\n(transduce #'pass (any? #'evenp) '(1 3 5 2 7 9))\n#+end_src\n\n#+RESULTS:\n: T\n\nUsage of =reduced= isn't limited to Transducers; Reducers can short-circuit the\nstream as well.\n\n#+begin_src lisp\n(defun any? (pred)\n  (lambda (\u0026optional (acc nil a-p) (input nil i-p))\n    (cond ((and a-p i-p)\n           ;; vvv (I) vvv\n           (if (funcall pred input)\n               (reduced t)\n               nil))\n           ;; ^^^ (I) ^^^\n          ((and a-p (not i-p)) acc) ;; (D)\n          (t nil))))\n#+end_src\n\nLike with =filter=, this Reducer requires a top-level predicate, so we add an\ninner =lambda=.\n\nWithin (I) we can see =reduced= employed. Seeing this, =transduce= will not continue\nand will instead go right to the (D) case. The final =acc= is =T=.\n\n** Sources\n\n=transduce= is a =defgeneric=, and so can be called on anything that has a\ncorresponding =defmethod= for it. There are many \"natural\" Sources like lists and\nvectors, but we can easily define our own and then supply a =transduce= method to\nadd it to the family of things we can iterate over.\n\nLet's review the =reversed= Source, a means by which to iterate over a vector in\nreverse order.\n\n#+begin_src lisp :exports both :results verbatim\n(in-package :transducers)\n(transduce #'pass #'string (reversed \"Hello\"))\n#+end_src\n\n#+RESULTS:\n: olleH\n\nIn order to have a distinct type to associate a =transduce= method with, we need a\nwrapper type:\n\n#+begin_src lisp\n(defstruct reversed\n  (vector #() :type cl:vector))\n#+end_src\n\nWe also supply a prettier constructor:\n\n#+begin_src lisp\n(defun reversed (vector)\n  \"Source: Yield a VECTOR's elements in reverse order.\"\n  (make-reversed :vector vector))\n#+end_src\n\nNow come a trio of functions that drive the iteration:\n\n#+begin_src lisp\n(defmethod transduce (xform f (source reversed))\n  (reversed-transduce xform f source))\n\n(defun reversed-transduce (xform f coll)\n  (let* ((init   (funcall f))  ;; (1) The (M) case of the Reducer.\n         (xf     (funcall xform f))  ;; (2) Putting the transducer/reducer chain together.\n         (result (reversed-reduce xf init coll)))  ;; (3) The work.\n    (funcall xf result)))  ;; (7) The (D) case of the Reducer.\n\n(defun reversed-reduce (f identity rev)\n  (let* ((vec (reversed-vector rev))\n         (len (length vec)))\n    ;; Simple recursion to drive the iteration.\n    (labels ((recurse (acc i)\n               (if (\u003c i 0)\n                   acc  ;; (4) We're done.\n                   (let ((acc (funcall f acc (aref vec i)))) ;; (5) Call the transducer chain.\n                     (if (reduced? acc)  ;; (6a) Short-circuiting occured. Time to go home.\n                         (reduced-val acc)\n                         (recurse acc (1- i))))))) ;; (6b) Otherwise, keep going.\n      (recurse identity (1- len)))))\n#+end_src\n\nAll types follow this pattern. In (1) and (2) we do initial setup. In (4) we've\nreached the natural end of the Source (e.g. the end of the vector).\n\n(5) is us actually calling the whole transducer chain.\n\nAt (6a) we see that we always need to check if the result of the current call\nwas \"reduced\", i.e. short-circuited.\n\nThat's it! The beauty of =defgeneric= is that its methods can be defined in other\nsystems. This is precisely how the extensions for =jzon=, =fset=, and [[https://codeberg.org/fosskers/nonempty/src/branch/master/src/transducers.lisp#L10-L11][nonempty]] are done.\n\n* Limitations\n\n1. This library is generally portable, but assumes your CL implementation\n   supports tail-call elimination within ~labels~.\n2. A way to model the common =zip= function has not yet been found, but I suspect\n   the answer lies in being able to pass multiple sources as ~\u0026rest~ arguments.\n\n* Resources\n\n- [[https://clojure.org/reference/transducers][Clojure: Transducers]]\n- [[https://clojure.org/guides/faq#transducers_vs_seqs][Clojure: What are good uses cases for transducers?]]\n- [[https://www.youtube.com/watch?v=4KqUvG8HPYo][Youtube: Inside Transducers]] (Rich Hickey)\n- [[https://codeberg.org/fosskers/transducers.el][Emacs Lisp: Transducers]]\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffosskers%2Ftransducers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffosskers%2Ftransducers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffosskers%2Ftransducers/lists"}