{"id":13463088,"url":"https://github.com/chadrem/workers","last_synced_at":"2025-05-16T11:03:46.915Z","repository":{"id":6344180,"uuid":"7580381","full_name":"chadrem/workers","owner":"chadrem","description":"Workers is a Ruby gem for performing work in background threads. Multi-threading made simple.","archived":false,"fork":false,"pushed_at":"2024-04-12T14:40:28.000Z","size":105,"stargazers_count":247,"open_issues_count":4,"forks_count":16,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-05-09T17:52:07.433Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://remesch.com/2013/01/22/workers-and-tribe-ruby-threading-simplified/","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chadrem.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-01-12T20:54:23.000Z","updated_at":"2025-02-28T17:42:27.000Z","dependencies_parsed_at":"2024-06-18T17:03:49.836Z","dependency_job_id":"59a7d6b6-7620-4b5a-b0a6-0afb7b3d1d9f","html_url":"https://github.com/chadrem/workers","commit_stats":{"total_commits":165,"total_committers":3,"mean_commits":55.0,"dds":"0.012121212121212088","last_synced_commit":"91a1a69d7a04e282c632378ad61b32db36731b30"},"previous_names":[],"tags_count":22,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadrem%2Fworkers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadrem%2Fworkers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadrem%2Fworkers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chadrem%2Fworkers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chadrem","download_url":"https://codeload.github.com/chadrem/workers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254518383,"owners_count":22084374,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T13:00:45.769Z","updated_at":"2025-05-16T11:03:46.864Z","avatar_url":"https://github.com/chadrem.png","language":"Ruby","readme":"# Workers [![Build Status](https://travis-ci.org/chadrem/workers.svg)](https://travis-ci.org/chadrem/workers) [![Coverage Status](https://coveralls.io/repos/chadrem/workers/badge.svg?branch=master\u0026service=github)](https://coveralls.io/github/chadrem/workers?branch=master)\n\nWorkers is a Ruby gem for performing work in background threads.\nDesign goals include high performance, low latency, simple API, customizability, and multi-layered architecture.\nIt provides a number of simple to use classes that solve a wide range of concurrency problems.\nIt is used by [Tribe](https://github.com/chadrem/tribe \"Tribe\") to implement event-driven actors.\n\n## Contents\n\n- [Installation](#installation)\n- [Parallel Map](#parallel-map)\n- [Tasks](#tasks)\n- [Workers](#workers)\n- [Pool](#pools)\n- [Timers](#timers)\n- [Schedulers](#schedulers)\n- [Concurrency and performance](#concurrency-and-performance)\n- [Contributing](#contributing)\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```Ruby\ngem 'workers'\n```\n\nAnd then execute:\n\n```\n$ bundle\n```\n\nOr install it yourself as:\n\n```\n$ gem install workers\n```\n\n## Parallel Map\n\nParallel map is the simplest way to get started with the Workers gem.\nIt is similar to Ruby's built-in Array#map method except each element is mapped in parallel.\n\n```Ruby\nWorkers.map([1, 2, 3, 4, 5]) { |i| i * i }\n```\n\nAny exceptions while mapping with cause the entire map method to fail.\nIf your block is prone to temporary failures (exceptions), you can retry it.\n\n```Ruby\nWorkers.map([1, 2, 3, 4, 5], :max_tries =\u003e 100) do |i|\n  if rand \u003c= 0.8\n    puts \"Sometimes I like to fail while computing #{i} * #{i}.\"\n    raise 'sad face'\n  end\n\n  i * i\nend\n```\n\n## Tasks\n\nTasks and task groups provide more flexibility than parallel map.\nFor example, you get to decide how you want to handle exceptions.\n\n```Ruby\n# Create a task group (it contains a pool of worker threads).\ngroup = Workers::TaskGroup.new\n\n# Add tasks to the group.\n10.times do |i|\n  10.times do |j|\n    group.add(:max_tries =\u003e 10) do\n      group.synchronize { puts \"Computing #{i} * #{j}...\" }\n      if rand \u003c= 0.9\n        group.synchronize { puts \"Sometimes I like to fail while computing #{i} * #{i}.\" }\n        raise 'sad face'\n      end\n      i * j # Last statement executed is the result of the task.\n    end\n  end\nend\n\n# Execute the tasks (blocks until the tasks complete).\n# Returns true if all tasks succeed.  False if any fail.\ngroup.run\n\n# Return an array of all the tasks.\ngroup.tasks\n\n# Return an array of the successful tasks.\ngroup.successes\n\n# Return an array of the failed tasks (raised an exception).\ngroup.failures\n\n# Review the results.\ngroup.tasks.each do |t|\n  t.succeeded? # True or false (false if an exception occurred).\n  t.failed?    # True or false (true if an exception occurred).\n  t.input      # Input value.\n  t.result     # Output value (the result of i * i in this example).\n  t.exception  # The exception if one exists.\n  t.max_tries  # Maximum number of attempts.\n  t.tries      # Actual number of attempts.\nend\n```\n\nNote that instances of TaskGroup provide a 'synchronize' method.\nThis method uses a mutex so you can serialize portions of your tasks that aren't thread safe.\n\n#### Options\n\n```Ruby\ngroup = Workers::TaskGroup.new(\n  :logger =\u003e nil,                   # Ruby logger instance.\n  :pool =\u003e Workers.pool             # The workers pool used to execute timer callbacks.\n)\n\ntask = Workers::Task.new(\n  :logger =\u003e nil,                   # Ruby logger instance.\n  :on_perform =\u003e proc {},           # Required option.  Block of code to run.\n  :input =\u003e [],                     # Array of arguments passed to the 'perform' block.\n  :on_finished =\u003e nil,              # Callback to execute after attempting to run the task.\n  :max_tries =\u003e 1,                  # Number of times to try completing the task (without an exception).\n)\n```\n\n## Workers\n\n#### Basic\n\nThe main purpose of the Worker class is to add an event system on top of Ruby's built-in Thread class.\nThis greatly simplifies inter-thread communication.\nYou must manually dispose of pools and workers so they get garbage collected.\n\n```Ruby\n# Initialize a worker pool.\npool = Workers::Pool.new(:on_exception =\u003e proc { |e|\n  puts \"A worker encountered an exception: #{e.class}: #{e.message}\"\n})\n\n# Perform some work in the background.\n100.times do\n  pool.perform do\n    sleep(rand(3))\n    raise 'sad face' if rand \u003c 0.5\n    puts \"Hello world from thread #{Thread.current.object_id}\"\n  end\nend\n\n# Wait up to 30 seconds for the workers to cleanly shutdown (or forcefully kill them).\npool.dispose(30) do\n  puts \"Worker thread #{Thread.current.object_id} is shutting down.\"\nend\n```\n\n#### Advanced\n\nThe Worker class is designed to be customized through inheritence and its event system:\n\n```Ruby\n# Create a subclass that handles custom events.\n# Super is called to handle built-in events such as perform and shutdown.\nclass CustomWorker \u003c Workers::Worker\n  private\n  def event_handler(event)\n    case event.command\n    when :my_custom\n      puts \"Worker received custom event: #{event.data}\"\n      sleep(1)\n    else\n      super(event)\n    end\n  end\nend\n\n# Create a pool that uses your custom worker class.\npool = Workers::Pool.new(:worker_class =\u003e CustomWorker, :on_exception =\u003e proc { |e|\n  puts \"A worker encountered an exception: #{e.class}: e.message}\"\n})\n\n# Tell the workers to do some work using custom events.\n100.times do |i|\n  pool.enqueue(:my_custom, i)\nend\n\n# Wait up to 30 seconds for the workers to cleanly shutdown (or forcefully kill them).\npool.dispose(30) do\n  puts \"Worker thread #{Thread.current.object_id} is shutting down.\"\nend\n```\n\n#### Without pools\n\nIn most cases you will be using a group of workers (a pool) as demonstrated above.\nIn certain cases, you may want to use a worker directly without the pool.\nThis gives you direct access to a single event-driven thread that won't die on an exception.\n\n```Ruby\n# Create a single worker.\nworker = Workers::Worker.new\n\n# Perform some work in the background.\n25.times do |i|\n  worker.perform do\n    sleep(0.1)\n    puts \"Hello world from thread #{Thread.current.object_id}\"\n  end\nend\n\n# Wait up to 30 seconds for the worker to cleanly shutdown (or forcefully kill it).\nworker.dispose(30)\n```\n\n#### Options\n\n```Ruby\nworker = Workers::Worker.new(\n  :logger =\u003e nil,                   # Ruby Logger instance.\n  :input_queue =\u003e nil,              # Ruby Queue used for input events.\n  :on_exception =\u003e nil              # Callback to execute on exception (exception passed as only argument).\n)\n```\n\n## Pools\n\nPools allow a group of workers to share a work queue.\nThe Workers gem has a default pool (Workers.pool) with 20 workers so in most cases you won't need to create your own.\nPools can be adjusted using the below methods:\n\n```Ruby\n# Create a pool.\npool = Workers::Pool.new\n\n# Return the number of workers in the pool.\npool.size\n\n# Increase the number of workers in the pool.\npool.expand(5)\n\n# Decrease the number of workers in the pool.\npool.contract(5)\n\n# Resize the pool size to a specific value.\npool.resize(20)\n```\n\n#### Options\n\n```Ruby\npool = Workers::Pool.new(\n  :size =\u003e 20,                      # Number of threads to create.\n  :logger =\u003e nil,                   # Ruby Logger instance.\n  :worker_class =\u003e Workers::Worker  # Class of worker to use for this pool.\n  :on_exception =\u003e nil              # Callback to execute on exception (exception passed as only argument).\n)\n```\n\n## Timers\n\nTimers provide a way to execute code in the future.\nYou can easily use them to tell a Worker or it's higher level classes (Task, TaskGroup, etc) to perform work in the future.\n\n```Ruby\n# Create a one shot timer that executes in 1.5 seconds.\ntimer = Workers::Timer.new(1.5) do\n  puts 'Hello world'\nend\n\n# Create a periodic timer that loops infinitely or until 'cancel' is called.\ntimer = Workers::PeriodicTimer.new(1) do\n  puts 'Hello world many times'\nend\n\n# Let the timer print some lines.\nsleep(5)\n\n# Shutdown the timer.\ntimer.cancel\n```\n\n#### Options\n\n```Ruby\ntimer = Workers::Timer.new(1,\n  :logger =\u003e nil,                   # Ruby logger instance.\n  :repeat =\u003e false,                 # Repeat the timer until 'cancel' is called.\n  :scheduler =\u003e Workers.scheduler,  # The scheduler that manages execution.\n  :callback =\u003e nil                  # The proc to execute (provide this or a block, but not both).\n)\n\ntimer = Workers::PeriodicTimer.new(1,\n  :logger =\u003e nil,                   # Ruby logger instance.\n  :scheduler =\u003e Workers.scheduler,  # The scheduler that manages execution.\n  :callback =\u003e nil                  # The proc to execute (provide this or a block, but not both).\n)\n```\n\n## Schedulers\n\nSchedulers are what trigger Timers to fire.\nThe Workers gem has a default scheduler (Workers.scheduler) so in most cases you won't need to create your own.\nSchedulers execute timers using a pool of workers so make sure your timer block is thread safe.\nYou can create additional schedulers as necessary:\n\n```Ruby\n# Create a workers pool with a larger than default thread count (optional).\npool = Workers::Pool.new(:size =\u003e 100)\n\n# Create a scheduler.\nscheduler = Workers::Scheduler.new(:pool =\u003e pool)\n\n# Create a timer that uses the above scheduler.\ntimer = Workers::Timer.new(1, :scheduler =\u003e scheduler) do\n  puts 'Hello world'\nend\n\n# Wait for the timer to fire.\nsleep(5)\n\n# Shutdown the scheduler.\nscheduler.dispose\n```\n\n#### Options\n\n```Ruby\nscheduler = Workers::Scheduler.new(\n  :logger =\u003e nil,                   # Ruby logger instance.\n  :pool =\u003e Workers::Pool.new        # The workers pool used to execute timer callbacks.\n)\n```\n\n#### Bucket Schedulers\n\nThe Bucket scheduler class is a specialized scheduler designed to work around lock contention.\nThis is accomplished by using many pools (100 by default) each with a small number of workers (1 by default).\nTimers are assigned to a scheduler by their ````hash```` value.\nMost users will never need to use this class, but it is documented here for completeness.\nBoth the number of buckets and the number of workers assigned to each bucket are configurable.\n\n```Ruby\n# Create a bucket scheduler.\nscheduler = Workers::BucketScheduler.new\n```\n\n#### Ruby's main thread\n\nRuby's main thread (the default thread) will terminate the Ruby process when it exits. Since asynchronouse code (such as the `Timer` and `Scheduler` classes) don't block the main thread, you may run into a problem where your script exits before your timers fire. If this is the case, you will need to provide a busy loop to keep the main thread from exiting:\n\n```Ruby\nwhile(true) do\n  sleep 1\nend\n```\n\n## Concurrency and performance\n\nWorkers is tested with both JRuby and MRI (C Ruby).\nBelow are some notes specific to each Ruby implementation.\nIn summary, JRuby is the recommended Ruby to use with Workers since it provides the highest performance with multiple CPU cores.\n\n#### JRuby (recommended)\n\nJRuby is designed for multi-threaded apps running on multiple cores.\nWhen used with Workers, you will be able to saturate all of your CPU cores with little to no tuning.\nIt is highly recommended you increase the number of workers in your pool if you have a large number of cores.\nA good starting point is 1x - 2x the number of cores for CPU bound apps.\nFor IO bound apps you will need to do some benchmarking to figure out what is best for you.\nA good starting point is 4x - 10x the number of cores for IO bound apps.\n\n#### MRI 2.2.0 or newer (supported)\n\nMRI 2.2.0 and above use real operating system threads with a global interpreter lock (GIL).\nThe bad news is that due to the GIL, only one thread can execute Ruby code at a given point in time.\nThis means your app will be CPU bound to a single core.\nThe good news is that IO bound applications will still see huge benefits from Workers.\nExamples of such IO are web service requests, web servers, web crawlers, database queries, writing to disk, etc.\nThreads performing such IO will temporarily release the GIL and thus let another thread execute Ruby code.\n\n#### MRI 1.8.x or older (not supported)\n\nThese old versions of Ruby use green threads (application layer threads) instead of operating system level threads.\nI recommend you upgrade to a newer version as I haven't tested Workers with them.\nThey also aren't officially supported by the Ruby community at this point.\n\n#### Rubinius\n\nI haven't tested Workers with Rubinius, but in theory it should just work.\nThe above JRuby notes should apply.\n\n## Contributing\n\n1. Fork it\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create new Pull Request\n","funding_links":[],"categories":["Developer Tools"],"sub_categories":["Concurrent Processing"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchadrem%2Fworkers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchadrem%2Fworkers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchadrem%2Fworkers/lists"}