{"id":13880289,"url":"https://github.com/tombenner/toro","last_synced_at":"2025-04-15T21:48:51.289Z","repository":{"id":59157801,"uuid":"19057411","full_name":"tombenner/toro","owner":"tombenner","description":"Multithreaded message processing on Postgres","archived":false,"fork":false,"pushed_at":"2015-02-01T01:35:24.000Z","size":822,"stargazers_count":37,"open_issues_count":0,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-15T21:48:45.072Z","etag":null,"topics":["background-jobs","message-queue","postgres","postgresql","queue","queueing","ruby","workers"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tombenner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"MIT-LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-04-23T04:51:35.000Z","updated_at":"2025-04-06T18:59:09.000Z","dependencies_parsed_at":"2022-09-13T17:52:00.345Z","dependency_job_id":null,"html_url":"https://github.com/tombenner/toro","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tombenner%2Ftoro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tombenner%2Ftoro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tombenner%2Ftoro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tombenner%2Ftoro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tombenner","download_url":"https://codeload.github.com/tombenner/toro/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249161104,"owners_count":21222468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["background-jobs","message-queue","postgres","postgresql","queue","queueing","ruby","workers"],"created_at":"2024-08-06T08:02:55.139Z","updated_at":"2025-04-15T21:48:51.267Z","avatar_url":"https://github.com/tombenner.png","language":"Ruby","readme":"Toro\n====\nTransparent, extensible background processing for Ruby \u0026 PostgreSQL\n\nOverview\n--------\n\nToro is a job queueing system (similar to Sidekiq or Resque) that runs on PostgreSQL and focuses on concurrency, visibility, extensibility, and durability:\n\n#### Concurrency\n* Toro can run many jobs simultaneously in a single process (a la Sidekiq; it uses [Celluloid](https://github.com/celluloid/celluloid))\n\n#### Visibility\n\nAn extensive dashboard:\n\n* Sort jobs by queue, worker, start time, queued time, duration, customizable name, status\n* Filter jobs by queue, worker, customizable name, status\n* Stacked histograms show the status distribution for each queue\n* A process table shows which machines/processes are active and which jobs they're running\n* Buttons for manually retrying failed jobs\n* Job detail view with in-depth job information:\n  * Basics: worker class, arguments, start time, duration, process host/PID\n  * Exception class, message, and backtrace of failed jobs\n  * A list of the exceptions and start times of retried jobs\n  * Customizable job properties\n\n#### Extensibility\n* Middleware support\n* Customizable UI views\n* Customizable job names\n* Customizable job properties\n  * Store job-related metadata that's set during the job's execution\n  * Stored in an hstore\n    * Properties can be indexed and queried against\n    * Jobs can be associated with other ActiveRecord models using a property as the foreign key\n\n#### Durability\n* Toro runs on PostgreSQL\n\n#### Other Features\n* Scheduled jobs\n* Configurable retry of failed jobs\n\n#### UI\n\nToro has an extensive dashboard that provides in-depth information about jobs, queues, processes, and more:\n\n[\u003cimg src=\"https://raw.github.com/tombenner/toro/master/examples/jobs.png\" width=\"48%\" /\u003e](https://raw.github.com/tombenner/toro/master/examples/jobs.png)\n[\u003cimg src=\"https://raw.github.com/tombenner/toro/master/examples/job.png\" width=\"48%\" /\u003e](https://raw.github.com/tombenner/toro/master/examples/job.png)\n[\u003cimg src=\"https://raw.github.com/tombenner/toro/master/examples/queues.png\" width=\"48%\" /\u003e](https://raw.github.com/tombenner/toro/master/examples/queues.png)\n[\u003cimg src=\"https://raw.github.com/tombenner/toro/master/examples/chart.png\" width=\"48%\" /\u003e](https://raw.github.com/tombenner/toro/master/examples/chart.png)\n\nInstallation\n------------\n\nAdd Toro to your Gemfile:\n\n```ruby\ngem 'toro'\n```\n\nMount the UI at a route in `routes.rb`:\n\n```ruby\nmount Toro::Monitor::Engine =\u003e '/toro'\n```\n\nAnd install and run the migration:\n\n```bash\nrails g toro:install\nrake db:migrate\n```\n\nQuick Start\n-----------\n\nCreate a worker:\n\n```ruby\n# app/workers/my_worker.rb\nclass MyWorker\n  include Toro::Worker\n\n  def perform(user_id)\n    puts \"Processing user #{user_id}...\"\n  end\nend\n```\n\nIn your controller action, model, or elsewhere, queue a job:\n```ruby\nMyWorker.perform_async(15)\n```\n\nStart Toro in the root directory of your Rails app:\n```bash\nrake toro\n```\n\nBasics\n------\n\n### Queues\n\nBy default, workers and processes use the `default` queue.\n\nTo set a worker's queue, use `toro_options`:\n\n```ruby\n# app/workers/my_worker.rb\nclass MyWorker\n  include Toro::Worker\n  toro_options queue: 'users'\n\n  def perform(user_id)\n    puts \"Processing user #{user_id}...\"\n  end\nend\n```\n\nTo set a process's queue, use `-q`:\n\n```bash\nrake toro -- -q users\n```\n\nOr specify multiple queues:\n\n```bash\nrake toro -- -q users -q comments\n```\n\n### Concurrency\n\nTo specify a process's concurrency (how many jobs it can run simultaneously), use `-c`:\n\n```bash\nrake toro -- -c 10\n```\n\n### Scheduled Jobs\n\nTo schedule a job for a specific time, use `perform_in(interval, *args)` or `perform_at(timestamp, *args)` instead of the standard `perform_async(*args)`:\n\n```ruby\nMyWorker.perform_in(2.hours, 'First arg', 'Second arg')\nMyWorker.perform_at(2.hours.from_now, 'First arg', 'Second arg')\n```\n\n### Retrying Jobs\n\nFailing jobs aren't retried by default. If you'd like Toro to retry a worker's failed jobs, specify the retry interval in the worker:\n\n```ruby\n# app/workers/my_worker.rb\nclass MyWorker\n  include Toro::Worker\n  toro_options retry_interval: 2.hours\n\n  def perform(user_id)\n    puts \"Processing user #{user_id}...\"\n  end\nend\n```\n\nThe error classes and times of retried jobs are stored as job properties.\n\n### Querying Jobs\n\n`Toro::Job` is an ActiveRecord model, which allows you to easily create complex queries against jobs that aren't easily performed in Redis-based job queueing systems. The model has the following columns:\n\n* queue - Queue\n* class_name - Worker class\n* args - Arguments\n* name - Name\n* created_at - When the job was created\n* scheduled_at - When the job was scheduled for (if it's a scheduled job)\n* started_at - When the job was started\n* finished_at - When the job finished (regardless of whether it succeeded or failed)\n* status - `queued`, `running`, `complete`, `failed`, or `scheduled`\n* started_by - Host and PID of the process running the job (e.g. `ip-10-55-10-151:1623`)\n* properties - An hstore containing customizable job properties\n\n\nJob Customization\n-----------------\n\n### Job Name\n\nTo set a job's name, define a `self.job_name` method that takes the same arguments the `perform` method:\n\n```ruby\nclass MyWorker\n  include Toro::Worker\n\n  def perform(user_id)\n  end\n\n  def self.job_name(user_id)\n    User.find(user_id).username\n  end\nend\n```\n\nA job name makes the job more recognizable in the UI. The UI also lets you search by name.\n\n### Job Properties\n\nJob properties let you store custom data about your jobs and their results.\n\nTo set job properties, make the `perform` method return a hash with a `:job_properties` key:\n\n```ruby\nclass MyWorker\n  include Toro::Worker\n\n  def perform(user_id)\n    comments = User.find(user_id).comments\n    # Do some processing...\n    {\n      job_properties: {\n        user_id: user_id,\n        comments_count: comments.length\n      }\n    }\n  end\nend\n```\n\nThe job properties will be shown in the job detail view in the UI.\n\nProperties are stored using [Nested Hstore](https://github.com/tombenner/nested-hstore), so you can store nested hashes, arrays, or any other types, allowing for NoSQL-like document storage:\n\n```ruby\nclass MyWorker\n  include Toro::Worker\n\n  def perform(user_id)\n    user = User.find(user_id)\n    comments = user.comments\n    # Do some processing...\n    {\n      job_properties: {\n        user: {\n          id: user.id,\n          is_blacklisted: user.is_blacklisted?,\n          timeline: {\n            is_private: user.timeline.is_private\n          }\n        },\n        comment_ids: comments.map(\u0026:id)\n      }\n    }\n  end\nend\n```\n\n#### Querying Job Properties\n\nJob properties are stored in an hstore, so you can query them (e.g. for reporting):\n\n```ruby\nbig_jobs = Toro::Job.where(\"(properties-\u003e'comments_count')::int \u003e ?\", 100)\n```\n\n#### Associating Jobs with Other Models\n\nYou can create associations between jobs and other models using job properties:\n\n```ruby\nclass User \u003c ActiveRecord::Base\n  has_many :jobs, foreign_key: \"toro_jobs.properties-\u003e'user_id'\", class_name: 'Toro::Job'\nend\n```\n\nYou can then, for example, find the failed jobs for a user:\n\n```ruby\nfailed_jobs = User.find(1).jobs.where(status: 'failed')\n```\n\nMiddleware\n----------\n\nToro's middleware support lets you run code \"around\" the processing of a job. Writing middleware is easy:\n\n```ruby\n# lib/my_middleware.rb\nclass MyMiddleware\n  def call(job, worker)\n    begin\n      puts \"Starting to process Job ##{job.id}\"\n      yield\n      puts \"Finished running Job ##{job.id}\"\n    rescue Exception =\u003e exception\n      puts \"Exception raised for Job ##{job.id}: #{exception}\"\n      job.update_attribute(status: 'failed')\n      raise exception\n    end\n  end\nend\n```\n\nThen register your middleware as part of the chain:\n\n```ruby\n# config/initializers/toro.rb\nToro.configure_server do |config|\n  config.server_middleware do |chain|\n    chain.add MyMiddleware\n  end\nend\n```\n\nToro supports the same server middleware inferface that Sidekiq does (including arguments, middleware removal, etc). Please see the [Sidekiq Middleware documentation](https://github.com/mperham/sidekiq/wiki/Middleware) for details.\n\nMonitor Customization\n---------------------\n\n### Chart\n\nA single histogram will be shown by default in the Chart view, but you can also split the queues into multiple histograms. (This is especially useful if you have a large number of queues and the single histogram has too many bars to be readable.) The keys of this hash are JS regex patterns for matching queues, and the values of the hash will be the titles of each histogram:\n\n```ruby\n# config/initializers/toro.rb\nToro::Monitor.options[:charts] = {\n  'ALL' =\u003e 'All',\n  'OTHER' =\u003e 'Default Priority',\n  '_high$' =\u003e 'High Priority',\n  '_low$' =\u003e 'Low Priority'\n}\n```\n\n`ALL` and `OTHER` are special keys: `ALL` will show all queues and `OTHER` will show all queues that aren't matched by the regex keys.\n\n### Poll Interval\n\nThe UI uses polling to update its data. By default, the polling interval is 3000ms, but you can adjust this like so:\n\n```ruby\n# config/initializers/toro.rb\nToro::Monitor.options[:poll_interval] = 5000\n```\n\n### Custom Job Views\n\nWhen you click on a job, a modal showing its properties is displayed. You can add subviews to this modal by creating a view in your app and calling `Toro::Monitor::CustomViews.add`, passing it the subview's title, the subview's filepath, and a block. The subview is only rendered if the block evaluates to true for the given job.\n\nIf you need to add JavaScript for the subview, you can do so by adding an asset path to `Toro::Monitor.options[:javascripts]`.\n\nFor example, the following code adds a subview that shows a \"Retry\" button for jobs with the specified statuses:\n\n```ruby\n# config/initializers/toro.rb\nview_path = Rails.root.join('app', 'views', 'toro', 'monitor', 'retry').to_s\nToro::Monitor::CustomViews.add('My View Title', view_path) do |job|\n  %w{complete failed}.include?(job.status)\nend\nToro::Monitor.options[:javascripts] \u003c\u003c 'toro/monitor/retry'\n```\n\n```\n/ app/views/toro/monitor/retry.slim\na class='btn btn-success' href='#' data-action='retry_job' data-job-id=job.id = 'Retry'\n```\n\n```coffee\n# app/assets/javascripts/toro/monitor/retry.js.coffee\n$ -\u003e\n  $('body').on 'click', '.job-modal [data-action=retry_job]', (e) -\u003e\n    id = $(e.target).attr('data-job-id')\n    $.get ToroMonitor.settings.api_url(\"jobs/retry/#{id}\")\n    alert 'Job has been retried'\n    false\n```\n\n### Authentication\n\nYou'll likely want to restrict access to the UI in a production environment. To do this, you can use routing constraints:\n\n#### Devise\n\nChecks a `User` model instance that responds to `admin?`\n\n```ruby\nconstraint = lambda { |request| request.env[\"warden\"].authenticate? and request.env['warden'].user.admin? }\nconstraints constraint do\n  mount Toro::Monitor::Engine =\u003e '/toro'\nend\n```\n\nAllow any authenticated `User`\n\n```ruby\nconstraint = lambda { |request| request.env['warden'].authenticate!({ scope: :user }) }\nconstraints constraint do\n  mount Toro::Monitor::Engine =\u003e '/toro'\nend\n```\n\nShort version\n\n```ruby\nauthenticate :user do\n  mount Toro::Monitor::Engine =\u003e '/toro'\nend\n```\n\n#### Authlogic\n\n```ruby\n# lib/admin_constraint.rb\nclass AdminConstraint\n  def matches?(request)\n    return false unless request.cookies['user_credentials'].present?\n    user = User.find_by_persistence_token(request.cookies['user_credentials'].split(':')[0])\n    user \u0026\u0026 user.admin?\n  end\nend\n\n# config/routes.rb\nrequire \"admin_constraint\"\nmount Toro::Monitor::Engine =\u003e '/toro', :constraints =\u003e AdminConstraint.new\n```\n\n#### Restful Authentication\n\nChecks a `User` model instance that responds to `admin?`\n\n```ruby\n# lib/admin_constraint.rb\nclass AdminConstraint\n  def matches?(request)\n    return false unless request.session[:user_id]\n    user = User.find request.session[:user_id]\n    user \u0026\u0026 user.admin?\n  end\nend\n\n# config/routes.rb\nrequire \"admin_constraint\"\nmount Toro::Monitor::Engine =\u003e '/toro', :constraints =\u003e AdminConstraint.new\n```\n\n#### Custom External Authentication\n\n```ruby\nclass AuthConstraint\n  def self.admin?(request)\n    return false unless (cookie = request.cookies['auth'])\n\n    Rails.cache.fetch(cookie['user'], :expires_in =\u003e 1.minute) do\n      auth_data = JSON.parse(Base64.decode64(cookie['data']))\n      response = HTTParty.post(Auth.validate_url, :query =\u003e auth_data)\n\n      response.code == 200 \u0026\u0026 JSON.parse(response.body)['roles'].to_a.include?('Admin')\n    end\n  end\nend\n\n# config/routes.rb\nconstraints lambda {|request| AuthConstraint.admin?(request) } do\n  mount Toro::Monitor::Engine =\u003e '/admin/toro'\nend\n```\n\n_(This authentication documentation was borrowed from the [Sidekiq wiki](https://github.com/mperham/sidekiq/wiki/Monitoring).)_\n\n\nLogging\n-------\n\nLogging can be especially useful in debugging concurrent systems like Toro. You can modify Toro's logger:\n\n\n```ruby\n# config/initializers/toro.rb\n\n# Adjust attributes of Toro's logger\nToro.logger.level = Logger::DEBUG\n\n# Or create a custom Logger\nToro.logger = Logger.new(Rails.root.join('log', 'toro.log'))\nToro.logger.level = Logger::DEBUG\n\n```\n\nSee the [Logger docs](http://www.ruby-doc.org/stdlib-2.0/libdoc/logger/rdoc/Logger.html) for more.\n\nTesting\n-------\n\nCopy and set up the database config:\n\n```bash\ncp spec/config/database.yml.example spec/config/database.yml\n```\n\nToro is tested against Rails 3 and 4, so please run the tests with [Appraisal](https://github.com/thoughtbot/appraisal) before submitting a PR. Thanks!\n\n```bash\nappraisal rspec\n```\n\nFAQ\n---\n\n### Toro?\n* [Toro](http://en.wikipedia.org/wiki/Tuna) is robust, quick, and values strength in numbers.\n* [Toro](http://en.wikipedia.org/wiki/Bull) is durable and runs a little large.\n* [Toro](http://en.wikipedia.org/wiki/T%C5%8Dr%C5%8D) values visibility.\n\n\nNotes\n-----\n\nA good deal of architecture and code was borrowed from [@mperham](https://github.com/mperham)'s excellent [Sidekiq](https://github.com/mperham/sidekiq), so many thanks to him and all of Sidekiq's contributors!\n\nLicense\n-------\n\nToro is released under the MIT License. Please see the MIT-LICENSE file for details.\n","funding_links":[],"categories":["Ruby"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftombenner%2Ftoro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftombenner%2Ftoro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftombenner%2Ftoro/lists"}