{"id":16613123,"url":"https://github.com/c-cube/moonpool","last_synced_at":"2025-03-15T12:30:28.383Z","repository":{"id":171820452,"uuid":"647552491","full_name":"c-cube/moonpool","owner":"c-cube","description":"Commodity thread pools and concurrency primitives for OCaml 5","archived":false,"fork":false,"pushed_at":"2025-02-21T19:08:00.000Z","size":23525,"stargazers_count":56,"open_issues_count":8,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-07T01:45:08.959Z","etag":null,"topics":["domains","futures","multicore","ocaml","ocaml5","thread-pool","work-stealing"],"latest_commit_sha":null,"homepage":"https://c-cube.github.io/moonpool/","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/c-cube.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-31T03:03:16.000Z","updated_at":"2025-02-21T19:02:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"ba2175c0-3c3b-4beb-b827-206e43f611c6","html_url":"https://github.com/c-cube/moonpool","commit_stats":null,"previous_names":["c-cube/moonpool"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-cube%2Fmoonpool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-cube%2Fmoonpool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-cube%2Fmoonpool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/c-cube%2Fmoonpool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/c-cube","download_url":"https://codeload.github.com/c-cube/moonpool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243730841,"owners_count":20338724,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["domains","futures","multicore","ocaml","ocaml5","thread-pool","work-stealing"],"created_at":"2024-10-12T01:46:02.564Z","updated_at":"2025-03-15T12:30:27.818Z","avatar_url":"https://github.com/c-cube.png","language":"OCaml","readme":"# Moonpool\n\n[![build](https://github.com/c-cube/moonpool/actions/workflows/main.yml/badge.svg)](https://github.com/c-cube/moonpool/actions/workflows/main.yml)\n\nA pool within a bigger pool (ie the ocean). Here, we're talking about\npools of `Thread.t` which live within a fixed pool of `Domain.t`.\n\nThis fixed pool of domains is shared between *all* the pools in moonpool.\nThe rationale is that we should not have more domains than cores, so\nit's easier to pre-allocate exactly that many domains, and run more flexible\nthread pools on top.\n\nIn addition, some concurrency and parallelism primitives are provided:\n- `Moonpool.Fut` provides futures/promises that execute\n    on these thread pools. The futures are thread safe.\n- `Moonpool.Chan` provides simple cooperative and thread-safe channels\n    to use within pool-bound tasks. They're essentially re-usable futures.\n\n    On OCaml 5 (meaning there's actual domains and effects, not just threads),\n    a `Fut.await` primitive is provided. It's simpler and more powerful\n    than the monadic combinators.\n- `Moonpool_forkjoin`, in the library `moonpool.forkjoin`\n    provides the fork-join parallelism primitives\n    to use within tasks running in the pool.\n\nOn OCaml 4.xx, there is only one domain; all threads run on it, but the\npool abstraction is still useful to provide preemptive concurrency.\n\n## Usage\n\nThe user can create several thread pools (implementing the interface `Runner.t`).\nThese pools use regular posix threads, but the threads are spread across\nmultiple domains (on OCaml 5), which enables parallelism.\n\nCurrent we provide these pool implementations:\n- `Fifo_pool` is a thread pool that uses a blocking queue to schedule tasks,\n    which means they're picked in the same order they've been scheduled (\"fifo\").\n    This pool is simple and will behave fine for coarse-granularity concurrency,\n    but will slow down under heavy contention.\n- `Ws_pool` is a work-stealing pool, where each thread has its own local queue\n    in addition to a global queue of tasks. This is efficient for workloads\n    with many short tasks that spawn other tasks, but the order in which\n    tasks are run is less predictable. This is useful when throughput is\n    the important thing to optimize.\n\nThe function `Runner.run_async pool task` schedules `task()` to run on one of\nthe workers of `pool`, as soon as one is available. No result is returned by `run_async`.\n\n```ocaml\n# #require \"threads\";;\n# let pool = Moonpool.Fifo_pool.create ~num_threads:4 ();;\nval pool : Moonpool.Runner.t = \u003cabstr\u003e\n\n# begin\n   Moonpool.Runner.run_async pool\n    (fun () -\u003e\n        Thread.delay 0.1;\n        print_endline \"running from the pool\");\n   print_endline \"running from the caller\";\n   Thread.delay 0.3; (* wait for task to run before returning *)\n  end ;;\nrunning from the caller\nrunning from the pool\n- : unit = ()\n```\n\nTo wait until the task is done, you can use `Runner.run_wait_block`[^1] instead:\n\n[^1]: beware of deadlock! See documentation for more details.\n\n```ocaml\n# begin\n   Moonpool.Runner.run_wait_block pool\n    (fun () -\u003e\n        Thread.delay 0.1;\n        print_endline \"running from the pool\");\n   print_endline \"running from the caller (after waiting)\";\n  end ;;\nrunning from the pool\nrunning from the caller (after waiting)\n- : unit = ()\n```\n\nThe function `Fut.spawn ~on f` schedules `f ()` on the pool `on`, and immediately\nreturns a _future_ which will eventually hold the result (or an exception).\n\nThe function `Fut.peek` will return the current value, or `None` if the future is\nstill not completed.\nThe functions `Fut.wait_block` and `Fut.wait_block_exn` will\nblock the current thread and wait for the future to complete.\nThere are some deadlock risks associated with careless use of these, so\nbe sure to consult the documentation of the `Fut` module.\n\n```ocaml\n# let fut = Moonpool.Fut.spawn ~on:pool\n    (fun () -\u003e\n       Thread.delay 0.5;\n       1+1);;\nval fut : int Moonpool.Fut.t = \u003cabstr\u003e\n\n# Moonpool.Fut.peek fut;\n- : int Moonpool.Fut.or_error option = None\n\n# Moonpool.Fut.wait_block_exn fut;;\n- : int = 2\n```\n\nSome combinators on futures are also provided, e.g. to wait for all futures in\nan array to complete:\n\n```ocaml\n# let rec fib x =\n    if x \u003c= 1 then 1 else fib (x-1) + fib (x-2);;\nval fib : int -\u003e int = \u003cfun\u003e\n\n# List.init 10 fib;;\n- : int list = [1; 1; 2; 3; 5; 8; 13; 21; 34; 55]\n\n# let fibs = Array.init 35 (fun n -\u003e Moonpool.Fut.spawn ~on:pool (fun () -\u003e fib n));;\nval fibs : int Moonpool.Fut.t array =\n  [|\u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e;\n    \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e;\n    \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e;\n    \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e;\n    \u003cabstr\u003e; \u003cabstr\u003e; \u003cabstr\u003e|]\n\n# Moonpool.Fut.join_array fibs |\u003e Moonpool.Fut.wait_block;;\n- : int array Moonpool.Fut.or_error =\nOk\n [|1; 1; 2; 3; 5; 8; 13; 21; 34; 55; 89; 144; 233; 377; 610; 987; 1597; 2584;\n   4181; 6765; 10946; 17711; 28657; 46368; 75025; 121393; 196418; 317811;\n   514229; 832040; 1346269; 2178309; 3524578; 5702887; 9227465|]\n```\n\n### Support for `await`\n\nOn OCaml 5, effect handlers can be used to implement `Fut.await : 'a Fut.t -\u003e 'a`.\n\nThe expression `Fut.await some_fut`, when run from inside some thread pool,\nsuspends its caller task; the suspended task is then parked, and will\nbe resumed when the future is completed.\nThe pool worker that was executing this expression, in the mean time, moves\non to another task.\nThis means that `await` is free of the deadlock risks associated with\n`Fut.wait_block`.\n\nIn the following example, we bypass the need for `Fut.join_array` by simply\nusing regular array functions along with `Fut.await`.\n\n```ocaml\n# let main_fut =\n    let open Moonpool.Fut in\n    spawn ~on:pool @@ fun () -\u003e\n    (* array of sub-futures *)\n    let tasks: _ Moonpool.Fut.t array = Array.init 100 (fun i -\u003e\n       spawn ~on:pool (fun () -\u003e\n           Thread.delay 0.01;\n           i+1))\n    in\n    Array.fold_left (fun n fut -\u003e n + await fut) 0 tasks\n  ;;\nval main_fut : int Moonpool.Fut.t = \u003cabstr\u003e\n\n# let expected_sum = Array.init 100 (fun i-\u003ei+1) |\u003e Array.fold_left (+) 0;;\nval expected_sum : int = 5050\n\n# assert (expected_sum = Moonpool.Fut.wait_block_exn main_fut);;\n- : unit = ()\n```\n\n### Errors\n\nWe have a `Exn_bt.t` type that comes in handy in many places. It bundles together\nan exception and the backtrace associated with the place the exception was caught.\n\n### Fibers\n\nOn OCaml 5, Moonpool comes with a library `moonpool.fib` (module `Moonpool_fib`)\nwhich provides _lightweight fibers_\nthat can run on any Moonpool runner.\nThese fibers are a sort of lightweight thread, dispatched on the runner's\nbackground thread(s).\nFibers rely on effects to implement `Fiber.await`, suspending themselves until the `await`-ed fiber\nis done.\n\n```ocaml\n# #require \"moonpool.fib\";;\n...\n\n# (* convenient alias *)\n  module F = Moonpool_fib;;\nmodule F = Moonpool_fib\n# F.main (fun _runner -\u003e\n    let f1 = F.spawn (fun () -\u003e fib 10) in\n    let f2 = F.spawn (fun () -\u003e fib 15) in\n    F.await f1 + F.await f2);;\n- : int = 1076\n```\n\nFibers form a _tree_, where a fiber calling `Fiber.spawn` to start a sub-fiber is\nthe sub-fiber's _parent_.\nWhen a parent fails, all its children are cancelled (forced to fail).\nThis is a simple form of [Structured Concurrency](https://en.wikipedia.org/wiki/Structured_concurrency).\n\nLike a future, a fiber eventually _resolves_ into a value (or an `Exn_bt.t`) that it's possible\nto `await`. With `Fiber.res : 'a Fiber.t -\u003e 'a Fut.t` it's possible to access that result\nas a regular future, too.\nHowever, this resolution is only done after all the children of the fiber have\nresolved — the lifetime of fibers forms a well-nested tree in that sense.\n\nWhen a fiber is suspended because it `await`s another fiber (or future), the scheduler's\nthread on which it was running becomes available again and can go on process another task.\nWhen the fiber resumes, it will automatically be re-scheduled on the same runner it started on.\nThis means fibers on pool P1 can await fibers from pool P2 and still be resumed on P1.\n\nIn addition to all that, fibers provide _fiber local storage_ (like thread-local storage, but per fiber).\nThis storage is inherited in `spawn` (as a shallow copy only — it's advisable to only\nput persistent data in storage to avoid confusing aliasing).\nThe storage is convenient for carrying around context for cross-cutting concerns such\nas logging or tracing (e.g. a log tag for the current user or request ID, or a tracing\nscope).\n\n### Fork-join\n\nOn OCaml 5, again using effect handlers, the sublibrary `moonpool.forkjoin`\nprovides a module `Moonpool_forkjoin`\nimplements the [fork-join model](https://en.wikipedia.org/wiki/Fork%E2%80%93join_model).\nIt must run on a pool (using `Runner.run_async` or inside a future via `Fut.spawn`).\n\nIt is generally better to use the work-stealing pool for workloads that rely on\nfork-join for better performance, because fork-join will tend to spawn lots of\nshorter tasks.\n\nHere is an simple example of a parallel sort.\nIt uses selection sort for small slices, like this:\n\n```ocaml\n# let rec select_sort arr i len =\n    if len \u003e= 2 then ( \n      let idx = ref i in\n      for j = i+1 to i+len-1 do\n        if arr.(j) \u003c arr.(!idx) then idx := j\n      done;\n      let tmp = arr.(!idx) in\n      arr.(!idx) \u003c- arr.(i);\n      arr.(i) \u003c- tmp;\n      select_sort arr (i+1) (len-1)\n    );;\nval select_sort : 'a array -\u003e int -\u003e int -\u003e unit = \u003cfun\u003e\n```\n\nAnd a parallel quicksort for larger slices:\n\n```ocaml\n# let rec quicksort arr i len : unit =\n    if len \u003c= 10 then select_sort arr i len\n    else (\n      let pivot = arr.(i + (len / 2)) in\n      let low = ref (i - 1) in\n      let high = ref (i + len) in\n\n      (* partition the array slice *)\n      while !low \u003c !high do\n        incr low;\n        decr high;\n        while arr.(!low) \u003c pivot do\n          incr low\n        done;\n        while arr.(!high) \u003e pivot do\n          decr high\n        done;\n        if !low \u003c !high then (\n          let tmp = arr.(!low) in\n          arr.(!low) \u003c- arr.(!high);\n          arr.(!high) \u003c- tmp\n        )\n      done;\n\n      (* sort lower half and upper half in parallel *)\n      Moonpool_forkjoin.both_ignore\n        (fun () -\u003e quicksort arr i (!low - i))\n        (fun () -\u003e quicksort arr !low (len - (!low - i)))\n    );;\nval quicksort : 'a array -\u003e int -\u003e int -\u003e unit = \u003cfun\u003e\n\n\n# let arr = [| 4;2;1;5;1;10;3 |];;\nval arr : int array = [|4; 2; 1; 5; 1; 10; 3|]\n# Moonpool.Fut.spawn\n    ~on:pool (fun () -\u003e quicksort arr 0 (Array.length arr))\n    |\u003e Moonpool.Fut.wait_block_exn;;\n- : unit = ()\n# arr;;\n- : int array = [|1; 1; 2; 3; 4; 5; 10|]\n\n\n# let arr =\n    let rand = Random.State.make [| 42 |] in\n    Array.init 40 (fun _-\u003e Random.State.int rand 300);;\nval arr : int array =\n  [|64; 220; 247; 196; 51; 186; 22; 106; 58; 58; 11; 161; 243; 111; 74; 109;\n    49; 135; 59; 192; 132; 38; 19; 44; 126; 147; 182; 83; 95; 231; 204; 121;\n    142; 255; 72; 85; 95; 93; 73; 202|]\n# Moonpool.Fut.spawn ~on:pool\n    (fun () -\u003e quicksort arr 0 (Array.length arr))\n  |\u003e Moonpool.Fut.wait_block_exn\n  ;;\n- : unit = ()\n# arr;;\n- : int array =\n[|11; 19; 22; 38; 44; 49; 51; 58; 58; 59; 64; 72; 73; 74; 83; 85; 93; 95; 95;\n  106; 109; 111; 121; 126; 132; 135; 142; 147; 161; 182; 186; 192; 196; 202;\n  204; 220; 231; 243; 247; 255|]\n```\n\nNote that the sort had to be started in a task (via `Moonpool.Fut.spawn`)\nso that fork-join would run on the thread pool.\nThis is necessary even for the initial iteration because fork-join\nrelies on OCaml 5's effects, meaning that the computation needs to run\ninside an effect handler provided by the thread pool.\n\n### More intuition\n\nTo quote [gasche](https://discuss.ocaml.org/t/ann-moonpool-0-1/12387/15):\n\n\u003cblockquote\u003e\nYou are assuming that, if pool P1 has 5000 tasks, and pool P2 has 10 other tasks, then these 10 tasks will get to run faster than if we just added them at the end of pool P1. This sounds like a “fairness” assumption: separate pools will get comparable shares of domain compute ressources, or at least no pool will be delayed too much from running their first tasks.\n\n[…]\n\n- each pool uses a fixed number of threads, all running simultaneously; if there are more tasks sent to the pool, they are delayed and will only get one of the pool threads when previous tasks have finished\n- separate pools run their separate threads simultaneously, so they compete for compute resources on their domain using OCaml’s systhreads scheduler – which does provide fairness in practice\n- as a result, running in a new pool enables quicker completion than adding to an existing pool (as we will be scheduled right away instead of waiting for previous tasks in our pool to free some threads)\n- the ratio of compute resources that each pool gets should be roughly proportional to its number of worker threads\n\u003c/blockquote\u003e\n\n## OCaml versions\n\nThis works for OCaml \u003e= 4.08.\n- On OCaml 4.xx, there are no domains, so this is just a library for regular thread pools\n    with not actual parallelism (except for threads that call C code that releases the runtime lock, that is).\n    C calls that do release the runtime lock (e.g. to call [Z3](https://github.com/Z3Prover/z3), hash a file, etc.)\n    will still run in parallel.\n- on OCaml 5.xx, there is a fixed pool of domains (using the recommended domain count).\n    These domains do not do much by themselves, but we schedule new threads on them, and form pools\n    of threads that contain threads from each domain.\n    Each domain might thus have multiple threads that belong to distinct pools (and several threads from\n    the same pool, too — this is useful for threads blocking on IO); Each pool will have threads\n    running on distinct domains, which enables parallelism.\n\n    A useful analogy is that each domain is a bit like a CPU core, and `Thread.t` is a logical thread running on a core.\n    Multiple threads have to share a single core and do not run in parallel on it[^2].\n    We can therefore build pools that spread their worker threads on multiple cores to enable parallelism within each pool.\n\nTODO: actually use https://github.com/haesbaert/ocaml-processor to pin domains to cores,\npossibly optionally using `select` in dune.\n\n## License\n\nMIT license.\n\n## Install\n\n```sh, skip\n$ opam install moonpool\n```\n\n[^2]: ignoring hyperthreading for the sake of the analogy.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-cube%2Fmoonpool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fc-cube%2Fmoonpool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fc-cube%2Fmoonpool/lists"}