{"id":32181634,"url":"https://github.com/bortexz/eventflow","last_synced_at":"2026-02-23T14:01:31.958Z","repository":{"id":56740196,"uuid":"524191212","full_name":"bortexz/eventflow","owner":"bortexz","description":"Event processing pipelines for Clojure","archived":false,"fork":false,"pushed_at":"2022-08-13T18:58:18.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-21T22:49:15.174Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"epl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bortexz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-12T18:41:13.000Z","updated_at":"2022-08-12T18:41:59.000Z","dependencies_parsed_at":"2022-08-16T01:10:10.170Z","dependency_job_id":null,"html_url":"https://github.com/bortexz/eventflow","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/bortexz/eventflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bortexz%2Feventflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bortexz%2Feventflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bortexz%2Feventflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bortexz%2Feventflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bortexz","download_url":"https://codeload.github.com/bortexz/eventflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bortexz%2Feventflow/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29745111,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-23T07:44:07.782Z","status":"ssl_error","status_checked_at":"2026-02-23T07:44:07.432Z","response_time":90,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-21T22:49:18.577Z","updated_at":"2026-02-23T14:01:31.953Z","avatar_url":"https://github.com/bortexz.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# eventflow\n\n[![Clojars Project](https://img.shields.io/clojars/v/io.github.bortexz/eventflow.svg)](https://clojars.org/io.github.bortexz/eventflow) [![Cljdoc](https://cljdoc.org/badge/io.github.bortexz/eventflow)](https://cljdoc.org/d/io.github.bortexz/eventflow)\n\nEvent processing pipelines for Clojure.\n\n## Install\n### Leiningen/Boot\n```clojure\n[io.github.bortexz/eventflow \"0.0.1\"]\n```\n\n### Clojure CLI/deps.edn\n```clojure \nio.github.bortexz/eventflow {:mvn/version \"0.0.1\"}\n```\n\n## Status\nAlpha. It currently works but hasn't yet been used in production. API may still undergo some breaking changes.\n\n## Features\n\n- Event processors are transducers that take events and (optionally) produce events.\n\n- **Incremental pipeline**, a pipeline implementation for controlled execution of events. Call `flush` to propagate the next event through it's processors (sequentially or in parallel), use `drain` to flush a pipeline until it has no more events left. Useful to run simulations or test your pipeline with a predefined set of events.\n    \n- **Async pipeline**, assembles a pipeline using core.async chans that asynchronously processes events. A chan can be published to the pipeline using `publish-ch`. Processor nodes can be added or removed anytime from the pipeline. For when you're ready to go realtime.\n\n## Creating nodes\nNodes are expressed as maps like:\n```clojure\n(require '[bortexz.eventflow :as ef])\n\n; Events shape is user defined, here a tuple [topic data] is used.\n{::ef/xform (map (fn [[topic data]]\n                   (case topic\n                     :input-1 [:output-1 data-1]\n                     :input-2 [:output-2 data-2])))\n ::ef/topics {:input-1 #{:output-1}\n              :input-2 #{:output-2}}\n ::ef/async {::ef/parallelism 8\n             ::ef/thread? true\n             ::ef/topic-buf-fn (fn [topic] (a/sliding-buffer 1))\n             ::ef/combine-chs-fn (fn [{:keys [input-1 input-2]}]\n                                   (a/merge [input-1 input-2]))}}\n```\n\n### Description\n- `::ef/xform` transducer fn that accepts events and may produce events.\n- `::ef/topics` map from subscribed topics to emitted topics. Currently emitted topics are not used/verified, and are merely used as metadata (e.g for printing a diagram of the pipeline)\n- `::ef/async` async properties of the node when used in an async pipeline:\n    - `::ef/parallelism` number of workers to consume events that this node is subscribed to.\n    - `::ef/thread?` when true, workers will use a real thread to process events. Otherwise, go-loops are used.\n    - `::ef/topic-buf-fn` 1-arity fn that returns a core.async buffer/int/nil added to chan of given topic.\n    - `::ef/combine-chs-fn` 1-arity fn that returns a core.async chan that combines each topic's ch to produce the events feeded into xform. Accepts a map `{\u003ctopic\u003e \u003cch\u003e}`, and defaults to `(core.async/merge (vals chs))`.\n\n## Building pipelines\nBoth pipelines accept `topic-fn` as first argument, and a map of pipeline-specific options.\n\n```clojure\n\n; This fn will work with both types of pipelines, so pipeline building can be reused across incremental/async pipelines\n(defn build-pipeline [p]\n  (-\u003e p \n      (add-node :node-1 {::ef/xform ...})\n      (add-node :node-2 {...})\n      (add-node :node-3 {...})))\n\n(def inc-p (ef/incremental-pipeline first {::ef/parallel? true}))\n\n(def async-p (ef/async-pipeline first \n                                {::ef/topic-buf-fn (fn [topic] (a/sliding-buffer 1))\n                                 ::ef/ex-handler (fn [ex] ...)}))\n\n(build-pipeline inc-p)\n(build-pipeline async-p)\n\n```\n\nOptions for each pipeline:\n- Incremental\n    - `::ef/parallel?` when true, processing nodes for next event will happen in parallel\n\n- Async\n    - `::ef/topic-buf-fn` returns async buffer/int/nil, to be used as buffer of each topics internal ch.\n    - `::ef/ex-handler` exception handler for exceptions happening inside node transducers. Defaults to use the uncaught exception handler of thread.\n\n## Executing pipelines\n\n### Incremental pipelines \n```clojure\n(defn inc-p (ef/incremental-pipeline first))\n\n;; Publish events to the pipeline. It doesn't execute nodes, only stores the events in an internal queue.\n(ef/publish inc-p [:topic-a])\n\n;; For processing the next event of the queue:\n(ef/flush inc-p) ; blocking until the event is processed by all nodes.\n\n;; You can get the current internal queue of events\n(ef/pending-events inc-p)\n\n;; Draining the pipeline will flush in loop until events queue is empty. Careful with cycles!\n(ef/drain inc-p) ; blocking until all events are processed.\n```\n\n### Async pipelines\n```clojure \n(defn inc-p (ef/incremental-pipeline first))\n\n;; Publish an event into the pipeline. If topic exists (i.e has any subscriber), then uses blocking \u003e!! \n;; to publish the event for asynchronous processing. \n(ef/publish inc-p [:topic-a])\n\n;; You can publish into an async pipeline from an async chan directly:\n(ef/publish-ch inc-p _chan-producing-events)\n\n```\n\n## Inspiration \nThanks to James Trunk for [this](https://www.youtube.com/watch?v=nt_cMdqqaPE) presentation, and to Tim Baldridge for [this](https://www.youtube.com/watch?v=096pIlA3GDo) one. Both helped me shape this library.\n\n## License\n\nCopyright © 2022 Alberto Fernandez\n\nDistributed under the Eclipse Public License version 1.0.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbortexz%2Feventflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbortexz%2Feventflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbortexz%2Feventflow/lists"}