{"id":32169486,"url":"https://github.com/casz92/poolder","last_synced_at":"2026-02-21T03:31:08.806Z","repository":{"id":299973852,"uuid":"1004777810","full_name":"casz92/poolder","owner":"casz92","description":"A compile-time builder that generates a concurrent pool of worker processes, batchers and schedulers for parallel task execution","archived":false,"fork":false,"pushed_at":"2025-12-11T18:52:54.000Z","size":67,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-30T16:34:55.799Z","etag":null,"topics":["parallel-computing","worker-pool","worker-threads"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/casz92.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-19T06:59:00.000Z","updated_at":"2025-12-11T18:52:58.000Z","dependencies_parsed_at":"2025-06-19T08:28:47.550Z","dependency_job_id":"fa8d6cb7-bba7-44bc-8f1c-a4f0623a80b6","html_url":"https://github.com/casz92/poolder","commit_stats":null,"previous_names":["casz92/poolder"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/casz92/poolder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casz92%2Fpoolder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casz92%2Fpoolder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casz92%2Fpoolder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casz92%2Fpoolder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/casz92","download_url":"https://codeload.github.com/casz92/poolder/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/casz92%2Fpoolder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29672704,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-21T03:11:15.450Z","status":"ssl_error","status_checked_at":"2026-02-21T03:10:34.920Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parallel-computing","worker-pool","worker-threads"],"created_at":"2025-10-21T16:03:06.318Z","updated_at":"2026-02-21T03:31:08.795Z","avatar_url":"https://github.com/casz92.png","language":"Elixir","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Poolder\n![Version](https://img.shields.io/badge/version-0.1.15-blue.svg)\n![Status](https://img.shields.io/badge/status-active-green.svg)\n![License](https://img.shields.io/badge/license-MIT-blue.svg)\n\nA compile-time builder that generates a concurrent pool of worker processes, batchers and schedulers for parallel task execution.\n\n### Contents\n- [Features](#key-features-of-poolder)\n- [Usage](#usage)\n  - [Building a pool of workers](#building-a-pool-of-workers)\n  - [Pool installation](#pool-installation)\n  - [Using pool](#using-pool)\n  - [Scheduling jobs](#scheduling-jobs)\n  - [Building a FactoryPool](#building-a-factorypool)\n  - [Building a batcher](#building-a-batcher)\n- [Installation](#installation)\n- [Behaviors](#callbacks)\n  - [Workers](#workers-callbacks)\n  - [Poolers](#pooler-callbacks)\n  - [Batchers](#batcher-callbacks)\n  - [Schedulers](#scheduler-callbacks)\n- [Testing](#testing)\n- [License](#license)\n- [Important Notice](#important-notice)\n\n## Key Features of Poolder\n- **Fixed Pool Size**: Defines a fixed-size pool of workers at compile-time, ideal for consistent and controlled workloads.\n- **Dynamic Pool Size**: Allows runtime scaling of workers based on system demands, enabling flexible throughput.\n- **Scheduled Tasks**: Supports compile-time definition of recurring jobs using cron-style expressions or time intervals.\n- **Runtime Rescheduling**: Create, reconfigure, or cancel scheduled tasks dynamically without requiring a system restart.\n- **Batchers**: Processes batches of messages with configurable size and timeout, **optimizing efficiency and throughput**.\n- **FactoryPool**: Manages multiple dynamic worker pools grouped logically. **Groups can be messaged collectively or individually**, and can be started, reconfigured, or stopped at runtime.\n- **Workers**: Lightweight and isolated processing units with custom callback support and built-in error handling.\n- **Tasker**: A concurrent task executor with controlled concurrency levels, perfect for keeping workloads responsive without overloading the system.\n\n\n## Usage\n\n### Building a pool of workers\n```elixir\ndefmodule MyWorker do\n  use Poolder.Worker,\n    # worker unique name\n    name: :myworker,\n    # retry options\n    retry: [count: 5, backoff: 1000],\n    # :low | :normal | :high | :max\n    priority: :normal,\n    # list of custom callbacks\n    callback: [\n      event: {EventBus, :notify},\n      push: {Phoenix.PubSub, :broadcast},\n      cast: {:websocket_client, :cast}\n    ]\n\n    @impl true\n    def handle_init(state) do\n      IO.puts(\"Worker started ##{inspect(state)} #{inspect(self())}\")\n      {:ok, state}\n    end\n\n    @impl true\n    def handle_job({:push, channel, message}, state) do\n      {:push, [channel, message], state}\n    end\n\n    def handle_job({:hardwork, message}, _state) do\n      heavy_work(message)\n    end\n\n    def handle_job({:hardwork, message, :notify}, state) do\n      result = heavy_work(message)\n\n      event_msg = %EventBus.Model.Event{\n        id: result.id,\n        topic: :done,\n        data: result\n      }\n\n      {:event, [event_msg], state}\n    end\n\n    def handle_job(:stop_worker, state) do\n      {:exit, state}\n    end\n\n    @impl true\n    # response updating state\n    def handle_call(:get_state, _from, state) do\n      {:set, state, state}\n    end\n\n    # response without updating state\n    def handle_call(:hardwork, _from, state) do\n      heavy_work(message)    \n    end\n\n    @impl true\n    def handle_error(_data, attempt, error, _stacktrace, state) do\n      IO.puts(\"Error: #{inspect(error)}\")\n\n      cond do\n        # retry immediately and update state\n        attempt == 1 -\u003e {:retry, %{state | key: 5}}\n        # backoff for 5 seconds\n        attempt == 2 -\u003e {:backoff, 5000}\n        # stop retrying\n        attempt == 4 -\u003e :halt\n        # retry immediately without updating state\n        true -\u003e :something\n      end\n    end\n\n    ## Private functions\n    defp heavy_work(message) do\n      # do something heavy\n      message\n    end\nend\n\n# Building a pool\ndefmodule MyPool do\n  use Poolder.Pooler,\n    # pool unique name\n    name: :mypool,\n    # initial number of workers\n    pool_size: 10,\n    # dynamic pool size, if true use: start_child/0, stop_child/0\n    dynamic: true,\n    # :round_robin | :random | :monotonic | :phash | :broadcast\n    mode: :round_robin,\n    # Poolder.Worker module (required)\n    worker: MyWorker\n\n  @impl true\n  def handle_init(sup_pid) do\n    IO.puts(\"Pool started ##{inspect(sup_pid)}\")\n  end\nend\n```\n\n### Pool installation\n```elixir\ndef MyApp.Application do\n  def start(_type, _args) do\n    children = [\n      MyPool,\n      MyBatcher,\n      Poolder.Tasker, [name: :mytasker, limit: 5, hibernate_after: 60_000],\n      MyPeriodicJobs,\n      WillyWonkaFactory\n    ]\n\n    opts = [strategy: :one_for_one, name: MyApp.Supervisor]\n    Supervisor.start_link(children, opts)\n  end\nend\n```\n\n### Using pool\n```elixir\n# cast to random worker\nmessage = \"Hello Worker!\"\nMyPool.cast({:push, \"lobby\", message})\n\n# cast to specific worker\nindex = 5\npid = MyPool.pid(index)\nMyPool.cast(pid, {:hardwork, message, :notify})\n\n# get pool size\nsize = MyPool.size()\n\nfor i \u003c- 0..size-1 do\n  pid = MyPool.pid(i)\n  MyPool.cast(pid, {:hardwork, message})\nend\n```\n\n### Scheduling jobs\n```elixir\ndefmodule MyPeriodicJobs do\n    use Poolder.Scheduler,\n    name: :myjobs,\n    jobs: [\n      # cron style or interval style\n      {:prune, \"*/15 * * * * *\"},\n      {:update, \"*/10 * * * * *\"},\n      {:five_seconds, 5_000}\n    ],\n    retry: [count: 5, backoff: 1000],\n    hibernate_after: 60_000,\n    # :low | :normal | :high | :max\n    priority: :low\n\n    def prune(_args) do\n      IO.puts \"Pruning\"\n    end\n\n    def update(_args) do\n      IO.puts \"Updating\"\n    end\n\n    def five_seconds(_args) do\n      # if return {:set, :timer.hours(2)} change scheduler interval\n      # if return {:set, :new_scheduler_name, \"0 0 0 * * *\"} create a new scheduled job\n      # if return :stop, stop all scheduled jobs\n      # if return :exit, stop current scheduled job\n      # any other return value continue the scheduled job\n      IO.puts \"Five seconds\"\n    end\n\n    @impl true\n    def handle_error(_job_name, _attempt, _error, _stacktrace, _state) do\n      IO.puts(\"Periodic job: #{inspect(event)}\")\n    end\n\n    @impl true\n    def handle_hibernate(state) do\n      IO.puts(\"Periodic job - handle_hibernate\")\n    end\nend\n```\n\n### Building a FactoryPool\n```elixir\ndefmodule WillyWonkaFactory do\n  use Poolder.FactoryPool,\n  # add this if you want to use GenServer like a child\n  caller: \u0026GenServer.call/3\nend\n\ndefmodule EchoWorker do\n  use GenServer\n\n  def start_link(state), do: GenServer.start_link(__MODULE__, state)\n\n  def init(state) do\n    # Monitor process, is not necesary if you use Poolder.Worker like as child\n    state.monitor.monitor(self())\n    {:ok, state}\n  end\n\n  def handle_call(:ping, _from, state), do: {:reply, :pong, state}\n  def handle_info({:set, new_state}, _state), do: {:noreply, new_state}\n  def handle_info(:print, state) do\n    IO.inspect(state)\n     {:noreply, state}\n  end\nend\n\n# usage\n{:ok, _factory} = WillyWonkaFactory.start_link()\n\nhersheys_group = :hersheys\nfeastables_group = :feastables\n\n{:ok, milton} = WillyWonkaFactory.start_child(hersheys_group, {EchoWorker, %{initial: true}})\n{:ok, mrbeast} = WillyWonkaFactory.start_child(feastables_group, {EchoWorker, %{initial: true}})\n{:ok, nolan} = WillyWonkaFactory.start_child(feastables_group, {EchoWorker, %{initial: true}})\nWillyWonkaFactory.count(feastables_group) == 2\n\n# Send a call\nWillyWonkaFactory.call(milton, :ping) == :pong\n\n# Send a cast\nWillyWonkaFactory.cast(milton,{:set, %{products: [\"Reese's\", \"Snickers\", \"KitKat\", \"Kisses\"]}})\n\n# Broadcast a message\nWillyWonkaFactory.broadcast(feastables_group, {:set, %{flavors: [\"Peanut Butter\", \"Milk Chocolate\", \"Cookies \u0026 creme\"]}})\n\n# Check the state of the workers\nWillyWonkaFactory.cast(milton, :print)\nWillyWonkaFactory.cast(mrbeast, :print)\nWillyWonkaFactory.cast(nolan, :print)\n```\n\n### Building a batcher\n```elixir\ndefmodule Batcher do\n  use Poolder.Batcher,\n    # batcher unique name\n    name: :mybatcher,\n    # batch size\n    limit: 100,\n    # batch timeout — flushes and sends the batch for processing after this time\n    # interval in milliseconds or :infinity\n    timeout: 10_000,\n    # reverse the batch order, fifo (default) or lifo (true)\n    reverse: false,\n    # retrive the batch as a stream or list\n    stream: true,\n    # result with indexs {index, value}\n    indexed: false,\n    # retry options\n    retry: [count: 3, backoff: 1000],\n    # hibernate after this time in milliseconds or :infinity\n    hibernate_after: 600_000,\n    # :low | :normal | :high | :max\n    priority: :high\n\n  @impl true\n  def handle_init(state) do\n    {:ok, state}\n  end\n\n  @impl true\n  def handle_batch(batch_stream, _state) do\n    batch = Enum.to_list(batch_stream)\n    IO.inspect(batch, label: \"batch\")\n  end\n\n  @impl true\n  def handle_hibernate(_state) do\n    IO.puts(\"hibernating\")\n  end\nend\n\n# usage\n{:ok, pid} = Batcher.start_link([])\nBatcher.push(pid, :yellow)\nBatcher.push(:mybatcher, 5)\nBatcher.push_front(pid, \"Amsterdam\")\nBatcher.push(pid, [\"red\", \"blue\", \"green\"])\nBatcher.pop_at(pid, 2)\nBatcher.push(pid, [\"orange\", \"apple\", \"watermelon\"])\nBatcher.flush(pid) # force processing if batch is not full\n```\n\n## Tasker\n```elixir\n{:ok, pid} = Tasker.start_link(name: :mytasker, limit: 2, hibernate_after: 60_000)\n\nresults =\n  for i \u003c- 1..100 do\n    fun = fn -\u003e\n      IO.inspect(i)\n    end\n\n    Tasker.execute(pid, fun)\n  end\n\nIO.inspect(results, label: \"results\")\n```\n\n## Installation\nAdd `poolder` to your list of dependencies in `mix.exs`:\n```elixir\ndef deps do\n  [\n    {:poolder, \"~\u003e 0.1.15\"}\n  ]\nend\n```\n\n## Callbacks\n### Workers callbacks\n- `handle_init/1`: Initializes the worker state.\n- `handle_job/2`: Handles the job execution.\n- `handle_call/3`: Handles synchronous calls.\n- `handle_hibernate/1`: Handles before hibernate.\n- `handle_error/5`: Handles job errors.\n\n### Pooler callbacks\n- `handle_init/1`: Handles pool ready.\n\n### Batcher callbacks\n- `handle_init/1`: Initializes the batcher state.\n- `handle_batch/2`: Handles the batch processing.\n- `handle_hibernate/1`: Handles before hibernate.\n\n### Scheduler callbacks\n- `handle_init/1`: Initializes the scheduler state.\n- `handle_job/2`: Handles the job execution.\n- `handle_error/5`: Handles job errors.\n- `handle_hibernate/1`: Handles before hibernate.\n\n\n## Testing\n```bash\nmix test test/pooler_test.exs\nmix test test/factory_pool_test.exs\nmix test test/batch_test.exs\nmix test test/scheduler_test.exs\nmix test test/tasker_test.exs\n```\n\n## License\nThis project is licensed under the MIT License.\n\n---\n## Important Notice\n\n ⚠️ This library is under active development and may undergo several changes before reaching version `0.2.0`. Some structures, functions, or behaviors may be modified or removed in future updates. If you plan to use it in production, it's recommended to pin a specific version or keep track of ongoing changes.\n\nThanks for being part of the journey! 🤗\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasz92%2Fpoolder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcasz92%2Fpoolder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcasz92%2Fpoolder/lists"}