https://github.com/mercadolibre/toiler
Toiler is a AWS SQS long-polling thread-based message processor.
https://github.com/mercadolibre/toiler
polling ruby sqs sqs-message sqs-poller threaded workers
Last synced: 4 months ago
JSON representation
Toiler is a AWS SQS long-polling thread-based message processor.
- Host: GitHub
- URL: https://github.com/mercadolibre/toiler
- Owner: mercadolibre
- License: lgpl-3.0
- Created: 2015-04-29T15:53:00.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2023-10-27T18:58:57.000Z (over 2 years ago)
- Last Synced: 2026-01-14T00:19:16.492Z (5 months ago)
- Topics: polling, ruby, sqs, sqs-message, sqs-poller, threaded, workers
- Language: Ruby
- Homepage:
- Size: 156 KB
- Stars: 17
- Watchers: 2
- Forks: 10
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Toiler
Toiler is a AWS SQS long-polling thread-based message processor.
It's based on [shoryuken](https://github.com/phstc/shoryuken) but takes
a different approach at loadbalancing and uses long-polling.
## Features
### Concurrency
Toiler allows to specify the amount of processors (threads) that should be spawned for each queue.
Instead of [shoryuken's](https://github.com/phstc/shoryuken) loadbalancing approach, Toiler delegates this work to the kernel scheduling threads.
### Long-Polling
A Fetcher thread is spawned for each queue.
Fetchers are resposible for polling SQS/PubSub and retreiving messages.
They are optimised to not bring more messages than the amount of processors avaiable for such queue.
By long-polling fetchers wait for a configurable amount of time for messages to become available on a single request, this prevents unneccesarilly requesting messages when there are none.
### Message Parsing
Workers can configure a parser Class or Proc to parse a message body before being processed.
### Deadline Extension
Toiler has the ability to automatically extend the ack deadline of and messages to prevent the message from re-entering the queue if processing of such message is taking longer than the queue's ack deadline or visibility timeout.
## Instalation
Add this line to your application's Gemfile:
```ruby
gem 'toiler'
```
And then execute:
$ bundle
Or install it yourself as:
$ gem install toiler
## Usage
### Worker class
```ruby
class MyWorker
include Toiler::Worker
toiler_options queue: 'default', concurrency: 5, auto_delete: true
toiler_options parser: :json
# toiler_options parser: ->(sqs_msg){ REXML::Document.new(sqs_msg.body) }
# toiler_options parser: MultiJson
# toiler_options deadline_extension: true
# toiler_options batch: true
# toiler_options queue: 'subscription', concurrency: 5, auto_delete: true, provider: :gcp
#Example connection client that should be shared across all instances of MyWorker
@@client = ConnectionClient.new
def initialize
@last_message = nil
end
def perform(sqs_msg, body)
#Workers are thread safe, yay!
#Each worker instance is assured to be processing only one message at a time
@last_message = sqs_msg
puts body
end
end
```
### Configuration
```yaml
aws:
access_key_id: ... # or <%= ENV['AWS_ACCESS_KEY_ID'] %>
secret_access_key: ... # or <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
region: us-east-1 # or <%= ENV['AWS_REGION'] %>
gcp:
project_id: my-project # or <%= ENV['GCP_PROJECT'] %>
credentials: /path/to/keyfile.json # or <%= ENV['GCP_CREDENTIALS'] %>
wait: 20 # The time in seconds to wait for messages during long-polling
```
### Rails Integration
You can tell Toiler to load your Rails application by passing the `-R` or `--rails` flag to the "toiler" command.
If you load Rails, and assuming your workers are located in the `app/workers` directory, they will be auto-loaded. This means you don't need to require them explicitly with `-r`.
### Start Toiler
```shell
bundle exec toiler -r worker.rb -C toiler.yml
```
Other options:
```bash
toiler --help
-d, --daemon Daemonize process
-r, --require [PATH|DIR] Location of the worker
-q, --queue QUEUE1,QUEUE2,... Queues to process
-C, --config PATH Path to YAML config file
-R, --rails Load Rails
-L, --logfile PATH Path to writable logfile
-P, --pidfile PATH Path to pidfile
-v, --verbose Print more verbose output
-h, --help Show help
```
## Credits
[Sebastian Schepens](https://github.com/sschepens) for the creation of the proyect.
But much of the credit goes to [Pablo Cantero](https://github.com/phstc), creator of [Shoryuken](https://github.com/phstc/shoryuken), and [everybody who contributed to it](https://github.com/phstc/shoryuken/graphs/contributors).
## Contributing
1. Fork it ( https://github.com/sschepens/toiler/fork )
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request