{"id":19719743,"url":"https://github.com/resque/resque-pool","last_synced_at":"2025-04-04T10:09:42.004Z","repository":{"id":944323,"uuid":"721535","full_name":"resque/resque-pool","owner":"resque","description":"quickly fork a pool resque workers, saving memory (w/REE) and monitoring their uptime.","archived":false,"fork":false,"pushed_at":"2023-09-04T15:50:10.000Z","size":433,"stargazers_count":451,"open_issues_count":40,"forks_count":149,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-04-12T20:28:20.290Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://rubygems.org/gems/resque-pool","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/resque.png","metadata":{"files":{"readme":"README.md","changelog":"Changelog.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2010-06-15T04:55:48.000Z","updated_at":"2024-05-13T14:59:28.313Z","dependencies_parsed_at":"2024-05-13T14:59:18.004Z","dependency_job_id":"2f29146c-5698-4065-a446-a6ab836b8165","html_url":"https://github.com/resque/resque-pool","commit_stats":{"total_commits":293,"total_committers":51,"mean_commits":5.745098039215686,"dds":"0.40955631399317405","last_synced_commit":"7886ddb0b127a8fc7fb243195f886062d693ccdd"},"previous_names":["nevans/resque-pool"],"tags_count":21,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/resque%2Fresque-pool","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/resque%2Fresque-pool/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/resque%2Fresque-pool/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/resque%2Fresque-pool/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/resque","download_url":"https://codeload.github.com/resque/resque-pool/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247157283,"owners_count":20893220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T23:09:11.016Z","updated_at":"2025-04-04T10:09:41.978Z","avatar_url":"https://github.com/resque.png","language":"Ruby","readme":"Resque Pool\n===========\n\n[![Build Status](https://travis-ci.org/nevans/resque-pool.png)](https://travis-ci.org/nevans/resque-pool)\n![Depfu](https://img.shields.io/depfu/nevans/resque-pool.svg)\n![Gem](https://img.shields.io/gem/v/resque-pool.svg)\n\nResque pool is a daemon for managing a pool of\n[resque](https://github.com/resque/resque) workers.  With a simple config file,\nit manages your workers for you, starting up the appropriate number of workers\nfor each worker type.\n\nBenefits\n---------\n\n* Less config - With a simple YAML file, you can start up a pool daemon, and it\n  will monitor your workers for you.\n* Less memory - If you are using ruby 2.0+ (with copy-on-write safe garbage\n  collection), this should save you a *lot* of memory when you are managing many\n  workers.\n* Faster startup - when you start many workers at once, they would normally\n  compete for CPU as they load their environments.  Resque-pool can load your\n  application once, then rapidly fork the workers after setup.  If a worker\n  crashes or is killed, a new worker will start up and take its place right away.\n\nUpgrading?\n-----------\n\nSee\n[Changelog.md](https://github.com/nevans/resque-pool/blob/master/Changelog.md)\nin case there are important or helpful changes.\n\nHow to use\n-----------\n\n### YAML file config\n\nCreate a `config/resque-pool.yml` (or `resque-pool.yml`) with your worker\ncounts.  The YAML file supports both using root level defaults as well as\nenvironment specific overrides (`RACK_ENV`, `RAILS_ENV`, and `RESQUE_ENV`\nenvironment variables can be used to determine environment).  For example in\n`config/resque-pool.yml`:\n\n```Yaml\nfoo: 1\nbar: 2\n\"foo,bar,baz\": 1\n\nproduction:\n  \"foo,bar,baz\": 4\n```\n\n### Rake task config\n\nRequire the rake tasks (`resque/pool/tasks`) in your `Rakefile`, load your\napplication environment, configure Resque as necessary, and configure\n`resque:pool:setup` to disconnect all open files and sockets in the pool\nmanager and reconnect in the workers.  For example, with rails you should put\nthe following into `lib/tasks/resque.rake`:\n\n```Ruby\nrequire 'resque/pool/tasks'\n\n# this task will get called before resque:pool:setup\n# and preload the rails environment in the pool manager\ntask \"resque:setup\" =\u003e :environment do\n  # generic worker setup, e.g. Hoptoad for failed jobs\nend\n\ntask \"resque:pool:setup\" do\n  # close any sockets or files in pool manager\n  ActiveRecord::Base.connection.disconnect!\n  # and re-open them in the resque worker parent\n  Resque::Pool.after_prefork do |job|\n    ActiveRecord::Base.establish_connection\n  end\nend\n```\n\n\nFor normal work with fresh resque and resque-scheduler gems add next lines in lib/rake/resque.rake\n\n```ruby\ntask \"resque:pool:setup\" do\n  Resque::Pool.after_prefork do |job|\n    Resque.redis.client.reconnect\n  end\nend\n```\n\n### Start the pool manager\n\nThen you can start the queues via:\n\n    resque-pool --daemon --environment production\n\nThis will start up seven worker processes, one exclusively for the foo queue,\ntwo exclusively for the bar queue, and four workers looking at all queues in\npriority.  With the config above, this is similar to if you ran the following:\n\n    rake resque:work RAILS_ENV=production QUEUES=foo \u0026\n    rake resque:work RAILS_ENV=production QUEUES=bar \u0026\n    rake resque:work RAILS_ENV=production QUEUES=bar \u0026\n    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz \u0026\n    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz \u0026\n    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz \u0026\n    rake resque:work RAILS_ENV=production QUEUES=foo,bar,baz \u0026\n\nThe pool manager will stay around monitoring the resque worker parents, giving\nthree levels: a single pool manager, many worker parents, and one worker child\nper worker (when the actual job is being processed).  For example, `ps -ef f |\ngrep [r]esque` (in Linux) might return something like the following:\n\n    resque    13858     1  0 13:44 ?        S      0:02 resque-pool-manager: managing [13867, 13875, 13871, 13872, 13868, 13870, 13876]\n    resque    13867 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Waiting for foo\n    resque    13868 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Waiting for bar\n    resque    13870 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Waiting for bar\n    resque    13871 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Waiting for foo,bar,baz\n    resque    13872 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Forked 7481 at 1280343254\n    resque     7481 13872  0 14:54 ?        S      0:00      \\_ resque-1.9.9: Processing foo since 1280343254\n    resque    13875 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Waiting for foo,bar,baz\n    resque    13876 13858  0 13:44 ?        S      0:00  \\_ resque-1.9.9: Forked 7485 at 1280343255\n    resque     7485 13876  0 14:54 ?        S      0:00      \\_ resque-1.9.9: Processing bar since 1280343254\n\nRunning as a daemon will default to placing the pidfile and logfiles in the\nconventional rails locations, although you can configure that.  See\n`resque-pool --help` for more options.\n\nSIGNALS\n-------\n\nThe pool manager responds to the following signals:\n\n* `HUP`   - reset config loader (reload the config file), reload logfiles, restart all workers.\n* `QUIT`  - gracefully shut down workers (via `QUIT`) and shutdown the manager\n  after all workers are done.\n* `INT`   - gracefully shut down workers (via `QUIT`) and immediately shutdown manager\n* `TERM`  - immediately shut down workers (via `INT`) and immediately shutdown manager\n  _(configurable via command line options)_\n* `WINCH` - _(only when running as a daemon)_ send `QUIT` to each worker, but\n  keep manager running (send `HUP` to reload config and restart workers)\n* `USR1`/`USR2`/`CONT` - pass the signal on to all worker parents (see Resque docs).\n\nUse `HUP` to help logrotate run smoothly and to change the number of workers\nper worker type.  Signals can be sent via the `kill` command, e.g.\n`kill -HUP $master_pid`\n\nIf the environment variable `TERM_CHILD` is set, `QUIT` and `TERM` will respond as\ndefined by Resque 1.22 and above. See http://hone.herokuapp.com/resque/2012/08/21/resque-signals.html\nfor details, overriding any command-line configuration for `TERM`. Setting `TERM_CHILD` tells\nus you know what you're doing.\n\nCustom Configuration Loader\n---------------------------\n\nIf the static YAML file configuration approach does not meet your needs, you can\nspecify a custom configuration loader.\n\nSet the `config_loader` class variable on Resque::Pool to an object that\nresponds to `#call` (which can simply be a lambda/Proc). The class attribute\nneeds to be set before starting the pool. This is usually accomplished\nin the `resque:pool:setup` rake task, as described above.\n\nFor example, if you wanted to vary the number of worker processes based on a\nvalue stored in Redis, you could do something like:\n\n```ruby\ntask \"resque:pool:setup\" do\n  Resque::Pool.config_loader = lambda do |env|\n    worker_count = Redis.current.get(\"pool_workers_#{env}\").to_i\n    {\"queueA,queueB\" =\u003e worker_count }\n  end\nend\n```\n\nIn order to query ActiveRecord, be sure establish the database connection:\n\n```ruby\ntask \"resque:pool:setup\" do                                           \n  Resque::Pool.config_loader = lambda do |env|\n    ActiveRecord::Base.establish_connection\n    JobQueue.pluck(:name, :workers).to_h\n  end\nend\n```\n\nThe configuration loader's `#call` method will be invoked about once a second.\nThis allows the configuration to constantly change, allowing you to scale the\nnumber of workers up or down based on different conditions.\nIf the response is generally static, the loader may want to cache the value it\nreturns. It can optionally implement a `#reset!` method, which will be invoked\nwhen the HUP signal is received, allowing the loader to flush its cache, or\nperform any other re-initialization.\n\nYou can reduce the frequency that your configuration loader is called by\nwrapping it with `Resque::Pool::ConfigLoaders::Throttled` and specifying a time\n(in seconds) to cache the previous value:\n\n```ruby\ntask \"resque:pool:setup\" do\n  redis_loader = lambda do |env|\n    worker_count = Redis.current.get(\"pool_workers_#{env}\").to_i\n    { \"queueA,queueB\" =\u003e worker_count }\n  end\n\n  # calls through to redis_loader at most once per 10 seconds\n  Resque::Pool.config_loader = Resque::Pool::ConfigLoaders::Throttled.new(\n    redis_loader, period: 10\n  )\nend\n```\n\nSee [the spec](spec/config_loaders/throttled_spec.rb) for more details.\n\nZero-downtime code deploys\n--------------------------\n\nIn a production environment you will likely want to manage the daemon using a\nprocess supervisor like `runit` or `god` or an init system like `systemd` or\n`upstart`.  Example configurations for some of these are included in the\n`examples` directory.  With these systems, `reload` typically sends a `HUP`\nsignal, which will reload the configuration but not application code.  The\nsimplest way to make workers pick up new code after a deploy is to stop and\nstart the daemon.  This will result in a period where new jobs are not being\nprocessed.\n\nTo avoid this downtime, leave the old pool running and start a new pool with\n`resque-pool --hot-swap`.\n\nThe `--hot-swap` flag will turn off pidfiles (so multiple pools can run at\nonce), enable a lock file (so your init system knows when the pool is running),\nand shut down other pools _after_ the workers have started for this pool.\nThese behaviors can also be configured separately (see `resque-pool --help`).\nThe `upstart` configs in the `examples` directory demonstrate how to supervise a\ndaemonized pool with hot swap.\n\nPlease be aware that this approach uses more memory than a simple restart, since\ntwo copies of the application code are loaded at once. _TODO: [#139](https://github.com/nevans/resque-pool/issues/139)_\n\nOther Features\n--------------\n\nYou can also start a pool manager via `rake resque:pool` or from a plain old\nruby script by calling `Resque::Pool.run`.\n\nWorkers will watch the pool manager, and gracefully shutdown (after completing\ntheir current job) if the manager process disappears before them.\n\nYou can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or\nwith the `--config` command line option.\n\nSee the `examples` directory for example `chef` cookbook and\n`god` config.  In the `chef` cookbook, you can also find example `init.d` and\n`muninrc` templates (all very out of date, pull requests welcome).\n\nTODO\n-----\n\nSee [the TODO list](https://github.com/nevans/resque-pool/issues) at github issues.\n\nContributors\n-------------\n\nSee [list of contributors on github](https://github.com/nevans/resque-pool/graphs/contributors) or [in the changelog](https://github.com/nevans/resque-pool/blob/master/Changelog.md)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresque%2Fresque-pool","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresque%2Fresque-pool","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresque%2Fresque-pool/lists"}