{"id":15540231,"url":"https://github.com/rm-hull/task-scheduler","last_synced_at":"2025-04-23T16:43:28.745Z","repository":{"id":62434522,"uuid":"69504404","full_name":"rm-hull/task-scheduler","owner":"rm-hull","description":"Fork/Join task scheduling in Clojure","archived":false,"fork":false,"pushed_at":"2023-11-30T00:21:09.000Z","size":75,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-03T12:17:27.765Z","etag":null,"topics":["clojure","concurrency","fork-join","macros","parallel-tasks","task-scheduler"],"latest_commit_sha":null,"homepage":"http://www.destructuring-bind.org/task-scheduler","language":"Clojure","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rm-hull.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-28T21:15:48.000Z","updated_at":"2024-01-09T14:31:25.000Z","dependencies_parsed_at":"2025-03-06T01:32:04.932Z","dependency_job_id":"ad849ac2-2b78-41cc-99fd-ff79bbc2e681","html_url":"https://github.com/rm-hull/task-scheduler","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rm-hull%2Ftask-scheduler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rm-hull%2Ftask-scheduler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rm-hull%2Ftask-scheduler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rm-hull%2Ftask-scheduler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rm-hull","download_url":"https://codeload.github.com/rm-hull/task-scheduler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250472131,"owners_count":21436084,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clojure","concurrency","fork-join","macros","parallel-tasks","task-scheduler"],"created_at":"2024-10-02T12:13:18.662Z","updated_at":"2025-04-23T16:43:28.725Z","avatar_url":"https://github.com/rm-hull.png","language":"Clojure","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Task Scheduler\n\n[![Build Status](https://github.com/rm-hull/task-scheduler/actions/workflows/clojure.yml/badge.svg)](https://github.com/rm-hull/task-scheduler/actions/workflows/clojure.yml)\n[![Coverage Status](https://coveralls.io/repos/rm-hull/task-scheduler/badge.svg?branch=main)](https://coveralls.io/r/rm-hull/task-scheduler?branch=main)\n[![Dependencies Status](https://versions.deps.co/rm-hull/task-scheduler/status.svg)](https://versions.deps.co/rm-hull/task-scheduler)\n[![Downloads](https://versions.deps.co/rm-hull/task-scheduler/downloads.svg)](https://versions.deps.co/rm-hull/task-scheduler)\n[![Clojars Project](https://img.shields.io/clojars/v/rm-hull/task-scheduler.svg)](https://clojars.org/rm-hull/task-scheduler)\n[![Maintenance](https://img.shields.io/maintenance/yes/2023.svg?maxAge=2592000)]()\n\nTask scheduler is a library designed to abstract over Java's [Fork/Join](https://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html)\nframework, in order to help make using lightweight fork/join tasks more idiomatic in Clojure.\nFork/Join was designed for tasks that can be recursively broken down into smaller\npieces. The inspiration and motivation for this library was the excellent Coursera\n[Parallel Programming in Scala](https://www.coursera.org/learn/parprog1/home/welcome) course run by _École Polytechnique Fédérale de Lausanne_.\n\nMostly, the implementation consists of wrapping the Java implementation, but\ncrucially includes a `fork` macro which accepts a body, converts this into a\nrecursive task which is then automatically submitted to the fork/join executor,\nto be run when there is available capacity. The result is available from the\nblocking `join` function.\n\n### Pre-requisites\n\nYou will need JDK8 and [Leiningen](https://github.com/technomancy/leiningen) 2.8.1 or above installed.\n\n### Building\n\nTo build and install the library locally, run:\n\n    $ cd task-scheduler\n    $ lein test\n    $ lein install\n\n### Including in your project\n\nThere is a version hosted at [Clojars](https://clojars.org/rm-hull/task-scheduler).\nFor leiningen include a dependency:\n\n```clojure\n[rm-hull/task-scheduler \"0.2.1\"]\n```\n\nFor maven-based projects, add the following to your `pom.xml`:\n\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003erm-hull\u003c/groupId\u003e\n  \u003cartifactId\u003etask-scheduler\u003c/artifactId\u003e\n  \u003cversion\u003e0.2.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\n#### Upgrading from 0.1.0\n\n- The `task` macro was renamed to `fork`.\n\n## API Documentation\n\nSee [www.destructuring-bind.org/task-scheduler](http://www.destructuring-bind.org/task-scheduler/) for API details.\n\n## Basic Usage\n\nThe first step for using the fork/join task-scheduler framework is to write\ncode that performs a segment of the work. Your code should look similar to the\nfollowing pseudocode:\n\n```\nif (my portion of the work is small enough)\n  do the work directly\nelse\n  split my work into two pieces\n  invoke the two pieces and wait for the results\n```\n\nWrap this code in a `fork` block, which will typically return a [RecursiveTask](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/RecursiveTask.html),\nhaving submitted it to a ForkJoinPool instance.\n\n### A dumb way to compute Fibonacci numbers\n\nAs a simple example of using the fork/join mechanism, what follows is a naïve\nFibonacci implementation to illustrate library use (which was more-or-less\nlifted straight from the java version in JDK8 docs), rather than an efficient\nalgorithm.\n\nSo, starting with a textbook recursive Fibonacci function:\n\n```clojure\n(defn fib [n]\n  (if (\u003c= n 1)\n    n\n    (let [f1 (fib (- n 1))\n          f2 (fib (- n 2))]\n\n      (+ f1 f2))))\n\n(map fib (range 1 20)))\n; =\u003e (1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181)\n```\n\nThe important thing to note here is that the calculation is split into two\nsub-calls (to `(fib (- n 1))` and `(fib (- n 2))`) which are performed\nrecursively until _n_ reaches 1. Large values of _n_ result in a combinatorial\nexplosion, but that is not of particular concern here.\n\nReferring to the pseudocode above, the only modification we need to in order to\nmake use of the fork/join task scheduler is to wrap the sub-calls, and then\nwait for the results. Compare the fork/join version:\n\n```clojure\n(use 'task-scheduler.core)\n\n(defn fib [n]\n  (if (\u003c= n 1)\n    n\n    (let [f1 (fork (fib (- n 1)))\n          f2 (fork (fib (- n 2)))]\n\n      (+ (join f1) (join f2)))))\n\n(map fib (range 1 20)))\n; =\u003e (1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181)\n```\n\nThe `fork` creates a new task for each sub-call (which would be executed in\nparallel), while the result is returned out of the `join`. The structure and\nflow of the two implementations is exactly the same.\n\nNote this particular implementation is likely to perform poorly because the\nsmallest subtasks are too small to be worthwhile splitting up. Instead, as is\nthe case for nearly all fork/join applications, you'd pick some minimum\ngranularity size (for example 10 here) for which you always sequentially solve\nrather than subdividing.\n\n#### Bootnote\n\nTypically, a fast linear algorithm for calculating Fibonacci numbers might be\nalong the lines of:\n\n```clojure\n(defn fib [a b]\n  (cons a (lazy-seq (fib b (+ a b)))))\n\n(take 10 (fib 1 1))\n; =\u003e (1 1 2 3 5 8 13 21 34 55)\n```\n\n### Parallel Sum\n\nAnother example (this time from [Dan Grossman](http://homes.cs.washington.edu/~djg/)'s _Parallelism and\nConcurrency_ course), converted from Java into Clojure, to sum up\n10-million integers:\n\n```clojure\n(def ^:dynamic *sequential-threshold* 5000)\n\n(defn sum\n  ([arr]\n   (sum arr 0 (count arr)))\n\n  ([arr lo hi]\n   (if (\u003c= (- hi lo) *sequential-threshold*)\n     (reduce + (subvec arr lo hi))\n     (let [mid (+ lo (quot (- hi lo) 2))\n           left (fork (sum arr lo mid))\n           right (fork (sum arr mid hi))]\n       (+ (join left) (join right))))))\n\n(def arr (vec (range 100000000)))\n\n(time (reduce + arr))\n;=\u003e \"Elapsed time: 317.234628 msecs\"\n;=\u003e 49999995000000\n\n(time (sum arr))\n;=\u003e \"Elapsed time: 186.297616 msecs\"\n;=\u003e 49999995000000\n```\n\nFor comparison, the parallel sum implementation is approximately twice as\nfast as `(reduce + arr)`, as measured on an Intel Core i5-5257U CPU @ 2.70GHz.\n\nSetting `*sequential-threshold*` to a good-in-practice value is a trade-off.\nThe documentation for the Fork/Join framework suggests creating parallel\nsubtasks until the number of basic computation steps is somewhere over 100 and\nless than 10,000. The exact number is not crucial provided you avoid extremes.\n\n#### Can we do better?\n\nLook carefully at the implementation of parallel sum: `left` and `right` are\nforked tasks, while the current thread is blocked waiting for them both to\nyield their results.\n\nThe revised version below still forks the `left` value, but `right` value is\ncomputed in-line, thereby eliminating creation of more parallel tasks than is\nnecessary; this is _slightly_ more efficient at the expense of seeming somewhat\nasymmetrical.\n\n```clojure\n(defn sum\n  ([arr]\n   (sum arr 0 (count arr)))\n\n  ([arr lo hi]\n   (if (\u003c= (- hi lo) *sequential-threshold*)\n     (reduce + (subvec arr lo hi))\n     (let [mid (+ lo (quot (- hi lo) 2))\n           left (fork (sum arr lo mid))\n           right (sum arr mid hi)]\n       (+ (join left) right)))))\n```\n\nHowever, the order is crucial, if the `left` had been joined before `right`\ninvoked, or `right` computed before `left` forked, then the entire\narray-summing algorithm would have no parallelism at all since each step would\ncompute sequentially.\n\nThis may've been important for JSR166/JDK6 \u0026 JDK7, but I believe that this is\nno longer the case for JDK8, so for the sake of avoiding the left/right\nordering 'gotcha', use fork in all cases.\n\n### Implicit Equations\n\nAny computation that is [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) can make use of the fork/join\nmechanism to split the work effort into disparate chunks. For example, the\n[implicit-equations](https://github.com/rm-hull/implicit-equations) project takes an equation of the form:\n\n```clojure\n(defn dizzy [x y]\n  (infix abs(sin(x ** 2 - y ** 2)) - (sin(x + y) + cos(x . y))))\n```\n\nand attempts to plot Cartesian coordinates where the value crosses zero, which\nresults in some spectacular charts:\n\n![PNG](https://rawgithub.com/rm-hull/implicit-equations/main/doc/dizzy.png)\n\nThere is no dependency, or need for communication (or synchronization needed) to\ncalculate the value for each point, and there is very little effort\nrequired to separate the problem into a number of parallel tasks. The tasks\nare split by bands, as illustrated [in the code](https://github.com/rm-hull/implicit-equations/blob/main/src/implicit_equations/plot.clj#L103-L104) and are joined before the function returns.\n\n## References\n\n- https://www.coursera.org/learn/parprog1/home/welcome\n- https://docs.oracle.com/javase/tutorial/essential/concurrency/forkjoin.html\n- https://github.com/rm-hull/implicit-equations\n- http://homes.cs.washington.edu/~djg/teachingMaterials/grossmanSPAC_forkJoinFramework.html#useful\n\n## License\n\nThe MIT License (MIT)\n\nCopyright (c) 2016 Richard Hull\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of\nthis software and associated documentation files (the \"Software\"), to deal in\nthe Software without restriction, including without limitation the rights to\nuse, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of\nthe Software, and to permit persons to whom the Software is furnished to do so,\nsubject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS\nFOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR\nCOPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER\nIN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN\nCONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frm-hull%2Ftask-scheduler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frm-hull%2Ftask-scheduler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frm-hull%2Ftask-scheduler/lists"}