{"id":29705765,"url":"https://github.com/bigbinary/sidekiq_sqs_processor","last_synced_at":"2026-02-06T12:37:37.769Z","repository":{"id":289093651,"uuid":"970098332","full_name":"bigbinary/sidekiq_sqs_processor","owner":"bigbinary","description":"A Ruby gem that integrates Amazon SQS with Sidekiq for efficient and scalable message processing.","archived":false,"fork":false,"pushed_at":"2025-07-17T09:48:16.000Z","size":199,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-07-23T15:13:57.249Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bigbinary.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-21T13:19:08.000Z","updated_at":"2025-07-17T09:48:36.000Z","dependencies_parsed_at":"2025-07-17T16:00:10.361Z","dependency_job_id":"f5e57c21-205f-4716-939a-2743b3175b70","html_url":"https://github.com/bigbinary/sidekiq_sqs_processor","commit_stats":null,"previous_names":["bigbinary/sidekiq_sqs_processor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/bigbinary/sidekiq_sqs_processor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigbinary%2Fsidekiq_sqs_processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigbinary%2Fsidekiq_sqs_processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigbinary%2Fsidekiq_sqs_processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigbinary%2Fsidekiq_sqs_processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bigbinary","download_url":"https://codeload.github.com/bigbinary/sidekiq_sqs_processor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bigbinary%2Fsidekiq_sqs_processor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29160816,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-06T07:18:23.844Z","status":"ssl_error","status_checked_at":"2026-02-06T07:13:32.659Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-23T15:08:47.411Z","updated_at":"2026-02-06T12:37:37.754Z","avatar_url":"https://github.com/bigbinary.png","language":"HTML","readme":"# Sidekiq SQS Processor\n\n[![Gem Version](https://badge.fury.io/rb/sidekiq_sqs_processor.svg)](https://badge.fury.io/rb/sidekiq_sqs_processor)\n\nA Ruby gem that integrates Amazon SQS with Sidekiq for efficient and scalable message processing. Process SQS messages asynchronously with the power and monitoring capabilities of Sidekiq.\n\n## Features\n\n- **Automatic SQS polling** - Continuously polls SQS queues for new messages\n- **Parallel processing** - Process messages across multiple Sidekiq workers\n- **SNS integration** - Auto-unwraps SNS notifications\n- **Configurable polling strategies** - Choose between continuous or scheduled polling\n- **Automatic retries** - Leverage Sidekiq's retry mechanism with SQS visibility timeout\n- **Easy message routing** - Route messages to different workers based on content\n- **Rails integration** - Simple setup with generators for Rails applications\n- **Comprehensive monitoring** - Track processing stats and worker health\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem 'sidekiq-sqs-processor'\n```\n\nAnd then execute:\n\n```bash\n$ bundle install\n```\n\nOr install it yourself:\n\n```bash\n$ gem install sidekiq_sqs_processor\n```\n\n## Dependencies\n\nThis gem requires the following dependencies in your application:\n\n- Sidekiq (~\u003e 7.0)\n- aws-sdk-sqs (~\u003e 1.0)\n- Rails (\u003e= 6.0)\n\n### Rails Setup\n\nIf you're using Rails, run the built-in generator to create the initializer and example worker:\n\n```bash\n$ rails generate sidekiq_sqs_processor:install\n```\n\n## Basic Configuration\n\n### Rails Applications\n\nAfter running the installer, edit the generated initializer at `config/initializers/sidekiq_sqs_processor.rb`:\n\n```ruby\nSidekiqSqsProcessor.configure do |config|\n  # AWS region for SQS queues\n  config.aws_region = 'us-east-1'\n\n  # SQS queue URLs to poll\n  config.queue_urls = [\n    'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'\n  ]\n\n  # Polling configuration\n  config.polling_type = :continuous\n\n  # Map queue URLs to worker classes\n  config.queue_workers = {\n    'https://sqs.us-east-1.amazonaws.com/123456789012/queue1' =\u003e 'Queue1Worker',\n    'https://sqs.us-east-1.amazonaws.com/123456789012/queue2' =\u003e 'Queue2Worker'\n  }\n\n  # Optional: Set custom logger\n  config.logger = Rails.logger\n\n  # Optional: Set custom error handler\n  config.error_handler = -\u003e(error, context) do\n    # Handle errors (e.g., report to monitoring service)\n    Rails.logger.error(\"SQS Error: #{error.message}\\nContext: #{context.inspect}\")\n  end\nend\n```\n\n### Non-Rails Applications\n\nFor non-Rails applications, add this to your initialization code:\n\n```ruby\nrequire 'sidekiq_sqs_processor'\n\nSidekiqSqsProcessor.configure do |config|\n  # AWS region for SQS queues\n  config.aws_region = 'us-east-1'\n\n  # SQS queue URLs to poll\n  config.queue_urls = [\n    'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'\n  ]\n\n  # Start polling when Sidekiq starts\n  Sidekiq.configure_server do |sidekiq_config|\n    sidekiq_config.on(:startup) do\n      SidekiqSqsProcessor.start_continuous_poller\n    end\n\n    sidekiq_config.on(:shutdown) do\n      SidekiqSqsProcessor.stop_continuous_poller\n    end\n  end\nend\n```\n\n## Creating Workers\n\n### Using the Generator (Rails)\n\n```bash\n$ rails generate sidekiq_sqs_processor:worker UserNotification\n```\n\nThis will create:\n- `app/workers/user_notification_worker.rb`\n- `spec/workers/user_notification_worker_spec.rb` (if using RSpec)\n- or `test/workers/user_notification_worker_test.rb` (if using Test::Unit)\n\n### Manual Worker Creation\n\nCreate a class that inherits from `SidekiqSqsProcessor::BaseWorker`:\n\n```ruby\nclass MyWorker \u003c SidekiqSqsProcessor::BaseWorker\n  # Override the process_message method to handle your SQS messages\n  def process_message(message_body)\n    # message_body is already parsed from JSON if it was a JSON string\n\n    # Your processing logic here\n    User.find_by(id: message_body['user_id'])\u0026.notify(message_body['message'])\n\n    # Return a result (optional)\n    { status: 'success' }\n  end\nend\n```\n\n## Usage Examples\n\n### Basic Message Processing\n\n```ruby\nclass OrderProcessor \u003c SidekiqSqsProcessor::BaseWorker\n  def process_message(message_body)\n    if message_body.is_a?(Hash) \u0026\u0026 message_body['order_id']\n      # Process an order\n      process_order(message_body['order_id'], message_body['items'])\n    else\n      logger.error(\"Invalid order message format: #{message_body.inspect}\")\n    end\n  end\n\n  private\n\n  def process_order(order_id, items)\n    # Your order processing logic\n    Order.process(order_id, items)\n  end\nend\n```\n\n### Handling Different Message Types\n\n```ruby\nclass EventProcessor \u003c SidekiqSqsProcessor::BaseWorker\n  def process_message(message_body)\n    case message_body['event_type']\n    when 'user_created'\n      create_user(message_body['data'])\n    when 'order_placed'\n      process_order(message_body['data'])\n    when 'payment_received'\n      process_payment(message_body['data'])\n    else\n      logger.warn(\"Unknown event type: #{message_body['event_type']}\")\n    end\n  end\n\n  # ... processing methods ...\nend\n```\n\n### Handling SNS Messages\n\nSNS messages are automatically detected and unwrapped, so the `message_body` will contain the inner message content:\n\n```ruby\nclass NotificationProcessor \u003c SidekiqSqsProcessor::BaseWorker\n  def process_message(message_body)\n    # For an SNS message, the SNS envelope has been removed\n    # and message_body is the parsed content of the SNS Message field\n\n    logger.info(\"Processing notification: #{message_body.inspect}\")\n    # ... your processing logic ...\n  end\nend\n```\n\n### Direct Message Enqueueing\n\nYou can also enqueue messages directly to a worker without going through SQS:\n\n```ruby\n# Enqueue a message to a specific worker\nSidekiqSqsProcessor.enqueue_message(\n  MyWorker,\n  { order_id: 123, items: ['item1', 'item2'] }\n)\n```\n\n## Configuration Options\n\n### AWS Configuration\n\n```ruby\nconfig.aws_region = 'us-east-1'              # AWS region\nconfig.aws_access_key_id = 'YOUR_KEY'        # Optional - uses env vars by default\nconfig.aws_secret_access_key = 'YOUR_SECRET' # Optional - uses env vars by default\n```\n\n### SQS Configuration\n\n```ruby\nconfig.queue_urls = ['https://sqs.region.amazonaws.com/account/queue']\nconfig.visibility_timeout = 300    # 5 minutes\nconfig.wait_time_seconds = 20      # 20 seconds (long polling)\nconfig.max_number_of_messages = 10 # Max messages per receive call\n```\n\n### Polling Configuration\n\n```ruby\nconfig.polling_type = :continuous  # :continuous or :scheduled\nconfig.polling_frequency = 60      # Only used for scheduled (seconds)\nconfig.polling_enabled = true      # Enable/disable polling\nconfig.poll_on_startup = true      # Start polling when app starts\n```\n\n### Sidekiq Configuration\n\n```ruby\nconfig.worker_queue_name = 'sqs_default' # Default Sidekiq queue\nconfig.worker_retry_count = 5            # Default retry count\n```\n\n### Error Handling\n\n```ruby\n# Custom error handler\nconfig.error_handler = -\u003e(error, context) do\n  # Send to your error tracking service\n  Sentry.capture_exception(error, extra: context)\nend\n```\n\n## Best Practices\n\n### Message Structure\n\nFor best routing and processing, use structured JSON messages:\n\n```json\n{\n  \"type\": \"order_placed\",\n  \"data\": {\n    \"order_id\": \"12345\",\n    \"customer_id\": \"67890\",\n    \"items\": [\n      {\"id\": \"item1\", \"quantity\": 2},\n      {\"id\": \"item2\", \"quantity\": 1}\n    ]\n  },\n  \"metadata\": {\n    \"source\": \"web\",\n    \"timestamp\": \"2025-04-21T10:15:30Z\"\n  }\n}\n```\n\n### Worker Organization\n\n- Create separate workers for different message types or domains\n- Use class inheritance for common processing logic\n- Keep workers small and focused\n\n### Error Handling\n\n- Use the `error_handler` config option for global error reporting\n- Implement custom error handling in your workers for specific cases\n- Let Sidekiq handle retries for transient failures\n\n### Visibility Timeout\n\n- Set your SQS visibility timeout longer than your Sidekiq job timeout\n- A good rule of thumb: visibility_timeout = (average_processing_time * 5) + max_retry_delay\n\n### Message Size Limits\n\nRemember that SQS has a 256KB message size limit. For larger data:\n- Store the data externally (e.g., S3) and include a reference in the message\n- Split large datasets across multiple messages\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`.\n\n## About BigBinary\n\n![BigBinary](https://raw.githubusercontent.com/bigbinary/bigbinary-assets/press-assets/PNG/logo-light-solid-small.png?raw=true)\n\nwheel is maintained by [BigBinary](https://www.bigbinary.com). BigBinary is a\nsoftware consultancy company. We build web and mobile applications using Ruby on\nRails, React.js and React Native.\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigbinary%2Fsidekiq_sqs_processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbigbinary%2Fsidekiq_sqs_processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbigbinary%2Fsidekiq_sqs_processor/lists"}