{"id":13509045,"url":"https://github.com/pragdave/work_queue","last_synced_at":"2025-10-21T18:42:06.839Z","repository":{"id":22119523,"uuid":"25450132","full_name":"pragdave/work_queue","owner":"pragdave","description":"Simple implementation of the hungry-consumer model in Elixir","archived":false,"fork":false,"pushed_at":"2020-03-08T01:18:01.000Z","size":24,"stargazers_count":41,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-22T04:47:55.002Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pragdave.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-10-20T04:00:39.000Z","updated_at":"2024-02-18T20:32:58.000Z","dependencies_parsed_at":"2022-08-20T20:40:41.566Z","dependency_job_id":null,"html_url":"https://github.com/pragdave/work_queue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pragdave%2Fwork_queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pragdave%2Fwork_queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pragdave%2Fwork_queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pragdave%2Fwork_queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pragdave","download_url":"https://codeload.github.com/pragdave/work_queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246323853,"owners_count":20759041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T02:01:02.255Z","updated_at":"2025-10-21T18:42:01.791Z","avatar_url":"https://github.com/pragdave.png","language":"Elixir","funding_links":[],"categories":["Queue"],"sub_categories":[],"readme":"WorkQueue\n=========\n\nA simple implementation of the _hungry consumer_ work scheduling model.\n\nGiven a queue of work to be processed, we create a a pool of workers.\nEach worker requests the next item to process from the queue. When it\nfinishes processing, it reports the result back and then requests the\nnext item of work.\n\nThe intent is that we do not need a central control point which\npreassigns work—that is done dynamically by the workers.\n\nThis has a couple of advantages. First, it locally maximizes CPU\nutilization, as workers are always busy as long as there is work in\nthe queue.\n\nSecond, it offers a degree of resilience, both against CPU hogging\nwork items and against worker failure.\n\nSimple Example\n--------------\n\n```elixir\nresults = WorkQueue.process(\n  fn (val,_) -\u003e { :ok, val*2 } end,   # worker function\n  [ 1, 2, 3 ]                         # work items to process\n)\n\nassert length(results) == 3\nfor {input, output} \u003c- results, do: assert(output == input * 2)\n```\n\nThis code will allocate a number of workers (the default is 2/3rds of\nthe available processing units in the node). Each worker then runs\nthe given function for as long as there are items to process. The\nresults are collected (in the order the workers return them) and\nreturned.\n\n\nThe API\n=======\n\n```elixir\nresults = WorkQueue.process(work_processor, item_source, options \\\\ [])\n```\n\n* `work_processor` is a function that transforms an item from the work\n    queue. It receives a value, and returns either `{:ok, result}` or\n  `{:error, reason}`\n\n* `item_source` is the source of the items to be processed. In the\n  simplest case, this is just an enumerable. Successive items are\n  taken from it and fed to the workers.\n\n  In other situations, you may need to perform additional processing\n  in order to generate the items. In particular, the item source may\n  be unbounded. In this case, you need to provide a `get_next_item`\n  function (using the options—see below). This function receives the\n  `item_source` as its initial state.\n\n* `options` is an optional keyword list:\n\n  * `worker_count: ` _count_\n\n     If count is an integer, start that number of workers. If it is a\n     float, it becomes a factor by which we multiply the number of\n     processing units on the node (so specifying `0.5` will start\n     workers for one-half the number of available processing units).\n     You can also pass the symbols `:cpu_bound` and `:io_bound`. The\n     former creates a worker for each processing unit, the latter\n     creates 10 workers per processing unit.\n\n  *  `get_next_item: ` _func_\n\n     The function that fetches the next item to be given to a worker.\n     It initially receives `item_source` as its parameter. It returns\n     a three element tuple. The first element is `:ok` if an item has\n     been returned, or `:done` otherwise. The second element is the\n     item to be returned, and the third is the updated item source\n     state.\n\n     The default value of `get_next_item` for list values of the item\n     source is\n\n  ```elixir\n     defp traverse_list([]),    do: {:done, nil, []}\n     defp traverse_list([h|t]), do: {:ok,   h,   t}\n  ```\n         \n  * `report_each_result_to: ` _func_\n\n     Invoke `func` as each result becomes available. The function\n     receives a tuple containing the original work item and the result\n     of running the calculation on that item. Its return value is\n     ignored.\n  \n  ```elixir\n     test \"notifications of results\" do\n       WorkQueue.process(\n         \u0026double/2,\n         [ 1, 2, 3 ],\n         report_each_result_to:\n           fn {input, output} -\u003e assert(output == input*2) end\n       )\n     end\n  ```\n         \n  * `report_progress_to:` _func_, `report_progress_interval:` _ms_\n\n     Invoke `func` to report progress. It is passed\n\n     * `{:starting}` when work is about to start\n     * `{:progress,` _count_`}` every `ms` milliseconds, indicating\n       the total number of items processed so far\n     * `{:finished,` _results_`}` reported when processing finishes\n\n     Progress reporting is disabled when `report_progress_interval` is\n     `false` (the default).\n     \n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpragdave%2Fwork_queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpragdave%2Fwork_queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpragdave%2Fwork_queue/lists"}