{"id":42790124,"url":"https://github.com/mercadolibre/toiler","last_synced_at":"2026-01-29T23:35:34.033Z","repository":{"id":31241814,"uuid":"34803250","full_name":"mercadolibre/toiler","owner":"mercadolibre","description":"Toiler is a AWS SQS long-polling thread-based message processor.","archived":false,"fork":false,"pushed_at":"2023-10-27T18:58:57.000Z","size":160,"stargazers_count":17,"open_issues_count":7,"forks_count":10,"subscribers_count":2,"default_branch":"master","last_synced_at":"2026-01-14T00:19:16.492Z","etag":null,"topics":["polling","ruby","sqs","sqs-message","sqs-poller","threaded","workers"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mercadolibre.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2015-04-29T15:53:00.000Z","updated_at":"2025-05-06T08:17:17.000Z","dependencies_parsed_at":"2023-07-15T15:46:06.346Z","dependency_job_id":null,"html_url":"https://github.com/mercadolibre/toiler","commit_stats":null,"previous_names":["sschepens/toiler"],"tags_count":22,"template":false,"template_full_name":null,"purl":"pkg:github/mercadolibre/toiler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mercadolibre%2Ftoiler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mercadolibre%2Ftoiler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mercadolibre%2Ftoiler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mercadolibre%2Ftoiler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mercadolibre","download_url":"https://codeload.github.com/mercadolibre/toiler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mercadolibre%2Ftoiler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28890445,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T21:06:44.224Z","status":"ssl_error","status_checked_at":"2026-01-29T21:06:42.160Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["polling","ruby","sqs","sqs-message","sqs-poller","threaded","workers"],"created_at":"2026-01-29T23:35:31.361Z","updated_at":"2026-01-29T23:35:34.019Z","avatar_url":"https://github.com/mercadolibre.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Toiler\n\nToiler is a AWS SQS long-polling thread-based message processor.\nIt's based on [shoryuken](https://github.com/phstc/shoryuken) but takes\na different approach at loadbalancing and uses long-polling.\n\n## Features\n\n### Concurrency\n\nToiler allows to specify the amount of processors (threads) that should be spawned for each queue.\nInstead of [shoryuken's](https://github.com/phstc/shoryuken) loadbalancing  approach, Toiler delegates this work to the kernel scheduling threads.\n\n### Long-Polling\n\nA Fetcher thread is spawned for each queue.\nFetchers are resposible for polling SQS/PubSub and retreiving messages.\nThey are optimised to not bring more messages than the amount of processors avaiable for such queue.\nBy long-polling fetchers wait for a configurable amount of time for messages to become available on a single request, this prevents unneccesarilly requesting messages when there are none.\n\n### Message Parsing\n\nWorkers can configure a parser Class or Proc to parse a message body before being processed.\n\n### Deadline Extension\n\nToiler has the ability to automatically extend the ack deadline of and messages to prevent the message from re-entering the queue if processing of such message is taking longer than the queue's ack deadline or visibility timeout.\n\n## Instalation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'toiler'\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install toiler\n\n## Usage\n\n### Worker class\n\n```ruby\nclass MyWorker\n  include Toiler::Worker\n\n  toiler_options queue: 'default', concurrency: 5, auto_delete: true\n  toiler_options parser: :json\n\n  # toiler_options parser: -\u003e(sqs_msg){ REXML::Document.new(sqs_msg.body) }\n  # toiler_options parser: MultiJson\n  # toiler_options deadline_extension: true\n  # toiler_options batch: true\n  # toiler_options queue: 'subscription', concurrency: 5, auto_delete: true, provider: :gcp\n\n  #Example connection client that should be shared across all instances of MyWorker\n  @@client = ConnectionClient.new\n    \n  def initialize\n    @last_message = nil\n  end\n\n  def perform(sqs_msg, body)\n    #Workers are thread safe, yay!\n    #Each worker instance is assured to be processing only one message at a time\n    @last_message = sqs_msg \n    puts body\n  end\nend\n```\n\n### Configuration\n\n```yaml\naws:\n  access_key_id:     ...             # or \u003c%= ENV['AWS_ACCESS_KEY_ID'] %\u003e\n  secret_access_key: ...             # or \u003c%= ENV['AWS_SECRET_ACCESS_KEY'] %\u003e\n  region:            us-east-1       # or \u003c%= ENV['AWS_REGION'] %\u003e\ngcp:\n  project_id:  my-project            # or \u003c%= ENV['GCP_PROJECT'] %\u003e\n  credentials: /path/to/keyfile.json # or \u003c%= ENV['GCP_CREDENTIALS'] %\u003e\nwait: 20                             # The time in seconds to wait for messages during long-polling\n```\n\n### Rails Integration\n\nYou can tell Toiler to load your Rails application by passing the `-R` or `--rails` flag to the \"toiler\" command.\n\nIf you load Rails, and assuming your workers are located in the `app/workers` directory, they will be auto-loaded. This means you don't need to require them explicitly with `-r`.\n\n\n### Start Toiler\n\n```shell\nbundle exec toiler -r worker.rb -C toiler.yml\n```\n\nOther options:\n\n```bash\ntoiler --help\n\n    -d, --daemon                     Daemonize process\n    -r, --require [PATH|DIR]         Location of the worker\n    -q, --queue QUEUE1,QUEUE2,...    Queues to process\n    -C, --config PATH                Path to YAML config file\n    -R, --rails                      Load Rails\n    -L, --logfile PATH               Path to writable logfile\n    -P, --pidfile PATH               Path to pidfile\n    -v, --verbose                    Print more verbose output\n    -h, --help                       Show help\n```\n\n\n## Credits\n[Sebastian Schepens](https://github.com/sschepens) for the creation of the proyect.\nBut much of the credit goes to [Pablo Cantero](https://github.com/phstc), creator of [Shoryuken](https://github.com/phstc/shoryuken), and [everybody who contributed to it](https://github.com/phstc/shoryuken/graphs/contributors).\n\n## Contributing\n\n1. Fork it ( https://github.com/sschepens/toiler/fork )\n2. Create your feature branch (`git checkout -b my-new-feature`)\n3. Commit your changes (`git commit -am 'Add some feature'`)\n4. Push to the branch (`git push origin my-new-feature`)\n5. Create a new Pull Request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmercadolibre%2Ftoiler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmercadolibre%2Ftoiler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmercadolibre%2Ftoiler/lists"}