{"id":39005975,"url":"https://github.com/thecsw/komi","last_synced_at":"2026-01-17T17:17:06.967Z","repository":{"id":143586262,"uuid":"616214736","full_name":"thecsw/komi","owner":"thecsw","description":"Komi - subarashii go pooling 🍡","archived":false,"fork":false,"pushed_at":"2025-04-25T01:03:30.000Z","size":937,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-25T02:20:37.390Z","etag":null,"topics":["golang","komi","parallel-computing","pooling"],"latest_commit_sha":null,"homepage":"https://sandyuraz.com/projects/komi","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thecsw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-19T23:25:13.000Z","updated_at":"2025-04-25T01:03:33.000Z","dependencies_parsed_at":null,"dependency_job_id":"2ecfe3e8-ee93-4f2b-9b87-1f8d93624e17","html_url":"https://github.com/thecsw/komi","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/thecsw/komi","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thecsw%2Fkomi","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thecsw%2Fkomi/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thecsw%2Fkomi/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thecsw%2Fkomi/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thecsw","download_url":"https://codeload.github.com/thecsw/komi/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thecsw%2Fkomi/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28512276,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T13:38:16.342Z","status":"ssl_error","status_checked_at":"2026-01-17T13:37:44.060Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","komi","parallel-computing","pooling"],"created_at":"2026-01-17T17:17:06.777Z","updated_at":"2026-01-17T17:17:06.945Z","avatar_url":"https://github.com/thecsw.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Komi - subarashi go pooling 🍡\n\n\u003cdiv id='badges' align='center'\u003e\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/thecsw/komi)](https://goreportcard.com/report/github.com/thecsw/komi)\n[![GoDoc](https://godoc.org/github.com/thecsw/komi?status.svg)](https://godoc.org/github.com/thecsw/komi)\n[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)\n\n\u003c/div\u003e\n\n![komi](komi.jpg)\n\n## Motivation\n\nGo is great for setting up easy parallel jobs and processes, however, it is not easy\nwhen one starts confusing concurrency with parallelism and ending up endlessly fighting\nrace conditions. `komi` is a generic pooling library that will satisfy your hunger.\n\n## Usage\n\nSay you want to run a function `foo(v)` that performs some kind of work on parameter `v`,\nbe it a database operation, a syscall, an IO operation, etc. (possibilities are endless!)\nSetting up a pool and sending jobs is as trivial as\n\n```go\npool := komi.New(komi.WorkSimple(foo))\ndefer pool.Close()\n// other code...\nerr = pool.Submit(v) // will block if pool is full\n```\n\nNotice that `pool.Close()` will gracefully free all the resources and channels occupied by\n`pool` by waiting for final jobs to complete. `pool.Close(true)` will force pool closure.\n\nBut what if you want to collect outputs of work performed on `v` with `foo(v) w`?\n\n```go\npool := komi.New(komi.Work(foo))\ndefer pool.Close()\n// collect outputs with pool.Outputs() channel\ngo func() {\n\t// Errors omitted for brevity\n\toutputs, _ := pool.Outputs()\n\tfor output := range outputs {\n\t\t// output is the result of `foo(v)`\n\t}\n}()\n// other code...\nerr = pool.Submit(v) // will block if pool is full\n```\n\nBut what if you want to collect errors as well? Consider `foo(v) error `\n\n```go\npool := komi.New(komi.WorkSimpleWithErrors(foo))\ndefer pool.Close()\n// collect errors with with pool.Errors() channel...\n// other code...\nerr = pool.Submit(v) // will block if pool is full\n```\n\nOr with `foo(v) (w, error)`!\n\n```go\npool := komi.New(komi.WorkWithErrors(foo))\ndefer pool.Close()\n// collect outputs with pool.Outputs() channel\n// collect errors with pool.Errors() channel...\n// other code...\nerr = pool.Submit(v) // will block if pool is full\n```\n\nSo, depending on what function you give, any work type is handled by the pool\non the fly! If work given doesn't produce outputs, `pool.Outputs()` will return `nil`,\nsimilarly, if work given doesn't produce errors, `pool.Errors()` will return `nil`.\n\nNote: if work produces outputs or errors, those activated channels **need** to be consumed\nby the user, otherwise, when reaching `size` number of elements in either (if active), work\nwill be blocked until the destination channel is consumed.\n\n## Connectors\n\nUnique feature of `komi` is that each pool can be connected with each other. Say you have two\nfunctions, where one opens file's contents, `openFile(filename string) (string, error)`,\nand the other counts the number of words, `countWords(contents string) int`.\n\nTwo pools can be created,\n\n```go\nopener := komi.NewWithSettings(komi.WorkWithErrors(openFile), \u0026komi.Settings{\n\tName:     \"Opener 📂 \",\n\tLaborers: 1,\n\tSize:     4,\n})\n\ncounter := komi.NewWithSettings(komi.Work(countWords), \u0026komi.Settings{\n\tName:     \"Counter 📚 \",\n\tLaborers: 10,\n\tSize:     20,\n})\n```\n\nWe can wire the outputs of `opener` to be automatically fed into `counter` with\n\n```go\nopener.Connect(counter)\n```\n\nSo now, those two pools are \"connected\". We would call this relationship as `opener` being\nthe dependent (child) pool and `counter` being the connected (parent) pool.\n\n```\n              Opener 📂                   Counter 📚\nfilenames  ┌─────────────┐  contents   ┌──────────────┐  word counts\n ───────\u003e  │  openFile   │  ────────\u003e  │  countWords  │  ────────\u003e\n .Submit   └─────────────┘  .Connect   └──────────────┘  .Outputs\n            pool: opener                 pool: counter \n```\n\n`komi`'s pools are smart enough that only by calling `counter.Close()`, it will issue a shutdown\ncommand back to `opener` and wait until it's closed. This closing logic procedure will happen with\nany number of connected pools.\n\nIf you have pools `1,2,...,N-1,N` connected in form `1-\u003e2-\u003e...-\u003eN-1-\u003eN`, user **can only call**\n`.Close()` on pool `N`, as it would be responsible for sending a closure request to `N-1`, and \nso on until `2` sends the shutdown request to `1`. When `1` is closed, the closure will resume\non `2`, up until `N-1` and `N`, where the latter will return from the original `Close()` call.\n\nPlease note that none of the pools `1,2,...,N-1` in the above will honor user's closure request,\nas it should come from their connected (parent) pool.\n\n## Quirks\n\nWhen the parent-most pool is closing, it will wait for all the child pools to complete their jobs.\nThis behavior can be overwritten by calling `.Close(true)`, which would skip any waiting of childrens'\nqueued jobs and skip any waiting of the parent-most pool's queued jobs.\n\n## Operations\n\nPools support waiting (blocking) until the pool has no jobs waiting for completion with `pool.Wait()`.\n\nSome other quality of life operations are also provided,\n\n- `Submit(v)` will submit job `v` to be performed by the pool. \n- `Close()` will close the pool if and only if it's disconnected or the parent-most pool.\n- `Close(true)` will close the pool ignoring any pending jobs.\n- `Outputs()` will return channel that the user should listen to for outputs (if work generated them).\n- `Errors()` will return channel that the user shoud listen to for errors (if work generates them).\n- `IsConnected()` will return true if the pool is a child of another pool, thus sending its outputs.\n- `IsClosed()` will return true if the pool has gracefully shutdown.\n- `JobsCompleted()` will return the number of jobs this pool has completed.\n- `JobsWaiting()` will return the number of jobs waiting in queue and currently in-work.\n- `JobsSucceeded()` will return the number of jobs completed with a non-nil errors.\n- `Name()` will return the pool's name (defaults to `Komi 🍡 `).\n\n## Settings\n\nYou can tune the performance and behavior of the pool with `komi.NewWithSetttings` by providing `*komi.Settings`,\n\n- `Laborers` sets the number of pool's laborers.\n- `Size` sets the size of the pool (how many jobs can wait until `pool.Submit` is blocked).\n- `Ratio` sets the `ratio` in `size = ratio * number of laborers` equation (only if size has not been manually set).\n- `LogLevel` sets the pool's logging level to `level`.\n- `Debug` sets the pool's logging level to `DebugLevel`.\n- `Name` sets the pool's name as shown in logs.\n\n## Stability\n\nThis is a brand new library I built for my [static website generator](https://github.com/thecsw/darkness),\nwhere it's used extensively and in production. However, there are no guarantees provided for this library,\nthat is, until something like `v1.0` is out, in which case, I would promise to maintain backward compatibility.\n\nPlease use it with knowing your risks. However, if you use a tagged version or a commit hash in your `go.mod`,\nyou should be fine.\n\n## Future work\n\nSome future items in mind:\n\n- Adding an error handler to `komi` polls, which if given, will be invoked if non-nil errors are returned by work.\n- More tests\n\n## Developers\n\nPlease consider giving it a try and filing an issue or a pull request.\n\nThank you!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthecsw%2Fkomi","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthecsw%2Fkomi","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthecsw%2Fkomi/lists"}