{"id":20492602,"url":"https://github.com/rafaelsales/smooth_queue","last_synced_at":"2026-04-21T10:06:33.343Z","repository":{"id":66167538,"uuid":"110379038","full_name":"rafaelsales/smooth_queue","owner":"rafaelsales","description":"Simple Redis message queue manager with flow control for Ruby","archived":false,"fork":false,"pushed_at":"2017-12-03T03:54:17.000Z","size":964,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-16T05:55:09.864Z","etag":null,"topics":["activejob","background-jobs","flow-control","message-queue","redis","resque","ruby","sidekiq"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rafaelsales.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-11T20:48:25.000Z","updated_at":"2018-05-13T19:37:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"3022839c-ad9b-4112-bbd9-ebb6e5602e2e","html_url":"https://github.com/rafaelsales/smooth_queue","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelsales%2Fsmooth_queue","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelsales%2Fsmooth_queue/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelsales%2Fsmooth_queue/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rafaelsales%2Fsmooth_queue/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rafaelsales","download_url":"https://codeload.github.com/rafaelsales/smooth_queue/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242073874,"owners_count":20067924,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activejob","background-jobs","flow-control","message-queue","redis","resque","ruby","sidekiq"],"created_at":"2024-11-15T17:29:48.211Z","updated_at":"2025-10-10T15:01:22.194Z","avatar_url":"https://github.com/rafaelsales.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Smooth Queue\n\nSimple Redis message queue manager with flow control for Ruby applications.\n\nSmooth Queue manages message queues in Redis making sure that your application don't try process more messages than it\ncan handle.\n\n# Motivation\n\nSometimes you don't want to process messages as fast as possible. Wat?\n\nA classic example is when you use a background processing framework (such as Resque or Sidekiq), and you don't want\ncertain workers to process more than 10 jobs simultaneously even though your background processor can handle more 50\njobs at a time.\n\n**Q:** Why would you not want to execute every single task in your application as fast as possible?\n\n**A:** Rate limiting is often a feature requirement/constraint. Here are a few examples:\n\n  1. The processing of certain messages are so expensive that can stress your application and cause slowness in the rest\n  of the application.\n\n  1. The processing of certain messages result in API calls, which target server imposes rate limiting.\n\n  1. You cannot afford computing power to handle excessive load when certain jobs are executed with high concurrency\n\n\u003cbr/\u003e\n\n**Q:** Can't background job libraries already do this?\n\n**A:** Kinda. They can, but none can do this efficiently, because their main focus is always get jobs done ASAP.\n\n  - Sidekiq: there are a couple options to achieve flow control, but none is super efficient:\n     - You can use custom job fetcher plugins, but then you're giving up on the extraordinary reliability that Mike\n     Perham has implemented for Sidekiq. This gets even worse if you pay for Sidekiq Pro, because it has even more\n     reliable fetch, but you would be overriding it\n     - If you pay for Sidekiq Enterprise, you can try to use the Limiter, but its algorithm is unfair and inneficient\n     as rate limiter\n\n   - Resque and Sidekiq: Create multiple queues and launch one Resque/Sidekiq process per queue that you want to limit,\n     so that you can specify the number of workers that will be available for that queue. There are two disadvantages\n     on this approach: (a) the more job types you need to limit, the more unnecessary memory usage the server will have,\n     for instance, 5 job types that need specific worker counts means 5 ruby processes; and (b) if you have server\n     redundancy, you're gonna have to devide the worker counts by the number of servers you have, which is an excessive\n     ops work\n\n# Usage\n\n### Configure when initializing the app:\n\n```ruby\nSmoothQueue.configure do |config|\n  config.add_queue('heavy_lifting', max_concurrency = 6) do |id, message|\n    # The id is generated by SmoothQueue to identify the message being processed\n    HeavyLiftingWorker.perform_later(id, message) # Make sure this operation is ~O(1)\n  end\n\n  config.add_queue('very_heavy_lifting', max_concurrency = 3) do |id, message|\n    # The id is generated by SmoothQueue to identify the message being processed\n    VeryHeavyLiftingWorker.perform_later(id, message) # Make sure this operation is ~O(1)\n  end\nend\n\nSmoothQueue.loop_async! # Starts a very lightweight thread that monitor the queues\n```\n\n### Process messages and tell SmoothQueue when it's done\n\n```ruby\nclass HeavyLiftingWorker\n  include SomeBackgroundJobFramework\n\n  def process(id, message)\n    # Make some heavy processing with the message { 'foo' =\u003e 'bar' }\n    SmoothQueue.done(id)\n  end\nend\n\nclass VeryHeavyLiftingWorker\n  include SomeBackgroundJobFramework\n\n  def process(id, message)\n    # Make some heavy processing with the message { 'bar' =\u003e 'baz' }\n    SmoothQueue.done(id)\n  end\nend\n```\n\n### Enqueue messages\n\n```ruby\nSmoothQueue.enqueue('heavy_lifting', { 'foo' =\u003e 'bar' })\nSmoothQueue.enqueue('very_heavy_lifting', { 'bar' =\u003e 'baz' })\n```\n\n# License\n\nPlease see [LICENSE](https://github.com/rafaelsales/smooth-queue/blob/master/LICENSE) for licensing details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaelsales%2Fsmooth_queue","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frafaelsales%2Fsmooth_queue","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frafaelsales%2Fsmooth_queue/lists"}