{"id":20863567,"url":"https://github.com/ezcater/sidekiq_publisher","last_synced_at":"2025-03-15T22:09:11.733Z","repository":{"id":32619723,"uuid":"132919305","full_name":"ezcater/sidekiq_publisher","owner":"ezcater","description":"Enqueue jobs to Sidekiq via a Postgres transaction","archived":false,"fork":false,"pushed_at":"2025-02-28T16:36:59.000Z","size":136,"stargazers_count":21,"open_issues_count":2,"forks_count":3,"subscribers_count":75,"default_branch":"main","last_synced_at":"2025-03-03T02:08:37.810Z","etag":null,"topics":["hacktoberfest","postgres","ruby","rubygems","sidekiq"],"latest_commit_sha":null,"homepage":"https://rubygems.org/gems/sidekiq_publisher","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ezcater.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-10T15:26:28.000Z","updated_at":"2025-02-28T16:37:01.000Z","dependencies_parsed_at":"2023-11-28T22:23:09.600Z","dependency_job_id":"e1d18c6c-e6a1-40af-823c-2213cc2a60f2","html_url":"https://github.com/ezcater/sidekiq_publisher","commit_stats":null,"previous_names":[],"tags_count":33,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezcater%2Fsidekiq_publisher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezcater%2Fsidekiq_publisher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezcater%2Fsidekiq_publisher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezcater%2Fsidekiq_publisher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ezcater","download_url":"https://codeload.github.com/ezcater/sidekiq_publisher/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243796722,"owners_count":20349263,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hacktoberfest","postgres","ruby","rubygems","sidekiq"],"created_at":"2024-11-18T05:29:43.742Z","updated_at":"2025-03-15T22:09:11.709Z","avatar_url":"https://github.com/ezcater.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sidekiq_publisher\n\n\u003e [!WARNING]\n\u003e This gem has been archived by ezCater and will no longer be receiving updates.\n\n[![Gem Version][gem_badge]][gem_link]\n\n[gem_badge]: https://badge.fury.io/rb/sidekiq_publisher.svg\n[gem_link]: https://badge.fury.io/rb/sidekiq_publisher\n\nThis gem provides support to enqueue jobs for Sidekiq by first staging the job\nin Postgres and relying on a separate process to communicate with Sidekiq/Redis.\n\nThe publisher process is alerted that a job is available to be published using\nPostgres NOTIFY/LISTEN.\n\nThis approach has the benefit that jobs can be published as part of a transaction\nthat modifies the system of record for the application. It also allows jobs to\nbe created even when Sidekiq/Redis is temporarily unavailable. The separate\npublisher process handles retries and ensure that each job is delivered to Sidekiq.\n\n\u003e :warning: Not all jobs are staged in Postgres. This is determined dynamically:\n\u003e if the job is enqueued from within an `ActiveRecord` transaction, then it is\n\u003e staged in Postgres. If not, then it bypasses Postgres and is enqueued directly\n\u003e to Redis via Sidekiq. To opt out of this behavior configure with\n\u003e SidekiqPublisher.configure { |c| c.stage_to_database_outside_transaction = true }\n\n## Installation\n\nAdd this line to your application's Gemfile:\n\n```ruby\ngem \"sidekiq_publisher\"\n```\n\nAnd then execute:\n\n    $ bundle\n\nOr install it yourself as:\n\n    $ gem install sidekiq_publisher\n\n\nRun the generator to create migrations for the jobs table and notifications:\n\n    $ rails generate sidekiq_publisher:install\n\n## Configuration\n\nThis gem uses the following configuration:\n\n* **logger**: the logger for this gem to use.\n* **exception_reporter**: a Proc that will be called with an exception\n* **metrics_reporter**: an optional object to record metrics. See below.\n* **batch_size**: the maximum number of jobs that will be enqueued together to Sidekiq\n* **job_retention_period**: the duration that published jobs will be kept in\n  Postgres after they have been enqueued to Sidekiq\n\n### Metrics Reporter\n\nThe metrics reporter that can be configured with an object that is expected to\nrespond to the following API:\n\n```ruby\ncount(metric_name, count)\ngauge(metric_name, count)\n```\n\nMetrics will be reported for:\n\n- the number of jobs published in each batch\n- the number of jobs purged\n\n#### Unpublished Jobs\n\nThere is also a module that can be used to record a metric for the number of\nunpublished jobs:\n\n```ruby\nSidekiqPublisher::ReportUnpublishedCount.call\n```\n\nIt is recommended to call this method periodically using something like\ncron or [clockwork](https://github.com/Rykian/clockwork).\n\n## Instrumentation\n\nInstrumentation of this library is implemented using\n[ActiveSupport::Notifications](https://api.rubyonrails.org/classes/ActiveSupport/Notifications.html).\n\nThe support for the configurable [metrics_reporter](lib/sidekiq_publisher/metrics_reporter.rb) and\n[exception_reporter](lib/sidekiq_publisher/exception_reporter.rb) options is implemented using\n[ActiveSupport::Subscriber](https://api.rubyonrails.org/classes/ActiveSupport/Subscriber.html).\n\nIf an alternate integration is required for metrics or error reporting then it can be implemented using outside this\nlibrary based on these examples.\n\n### Tracing\n\nThe instrumentation in the library also supports integration with application tracing products, such as\n[Datadog APM](https://www.datadoghq.com/product/apm/).\n\nThere is an optional integration with Datadog APM that can be required:\n\n```ruby\nrequire \"sidekiq_publisher/datadog_apm\"\n```\n\nThis file must be required in addition including the `sidekiq_publisher` gem or requiring `sidekiq_publisher`.\n\nThis integration covers all of the sections of the library that are instrumented and serves an\n[example](lib/sidekiq_publisher/datadog_apm) for implementing trace reporting for other products outside this library.\n\n## Usage\n\n### ActiveJob Adapter\n\nThis gem includes an adapter to use `SidekiqPublisher` with `ActiveJob`. This\nadapter must be explicitly required:\n\n```ruby\nrequire \"active_job/queue_adapters/sidekiq_publisher_adapter\"\n```\n\nThe adapter can also be required via your Gemfile:\n\n```ruby\ngem \"sidekiq_publisher\", require: [\"sidekiq_publisher\", \"active_job/queue_adapters/sidekiq_publisher_adapter\"]\n```\n\nThe adapter to use with `ActiveJob` must be specified in Rails configuration\n\n```ruby\n# application.rb\nconfig.active_job.queue_adapter = :sidekiq_publisher\n\n# or directly in configuration\nRails.application.config.active_job.queue_adapter = :sidekiq_publisher\n```\n\nTo selectively roll out the `SidekiqPublisher`, the adapter can be overridden for\na specific job class:\n\n```ruby\nclass MyJob \u003c ApplicationJob\n  self.queue_adapter = :sidekiq_publisher\nend\n```\n\n#### ActiveJob Exception Reporting\n\nMany exception monitoring service (e.g. Sentry, Airbrake, Honeybadger, etc) already provide basic integration support for `Sidekiq`.\nThese integration should also work with `SidekiqPublisher`.\nHowever, you may need to explicitly include\n`ActiveJob::QueueAdapters::SidekiqPublisherAdapter` as a compatible adapter for this to work properly.\n\nAlternatively, you can manually report the exception:\n\n ```ruby\nretry_on SomeError, attempts: 10 do |_job, exception|\n  Raven.capture_exception(exception, extra: { custom: :foo }) # Reporting using the Sentry gem\nend\n```\n\n### SidekiqPublisher::Worker\n\nSidekiq workers are usually defined by including `Sidekiq::Job` or\n`Sidekiq::Worker` in a class.\n\nTo use the `SidekiqPublisher`, this can be replaced by including\n`SidekiqPublisher::Worker`. The usual `perform_async`, etc methods will be\navailable on the class but jobs will be staged in the Postgres table.\n\n### Tying to a transaction\nTo guarantee that your job is enqueued when there's a change to the\nsystem of record, simply publish it during the transaction\nrepresenting that change. Usually, that can be accomplished by\npublishing in one of the ActiveRecord callbacks that are called\nwithin-transaction (e.g. `after_save`, but not `after_commit` and its\nderivatives):\n\n```ruby\nclass Frob \u003c ApplicationRecord\n  after_save do\n    MyJob.perform_later id\n  end\nend\n```\n\nFor considering more complicated situations (e.g. jobs that should be\nguaranteed during specific changes across models), the rails guides on\n[querying](https://guides.rubyonrails.org/active_record_querying.html)\nand\n[callbacks](https://guides.rubyonrails.org/active_record_callbacks.html),\nand the documentation on\n[transactions](https://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html)\nin ActiveRecord are good resources to consult.\n\n### Running\n\nThe publisher process that pulls the job data from Postgres and puts them into Redis\ncan be run with a rake task that is added via Railtie for Rails applications:\n\n```bash\nbundle exec rake sidekiq_publisher:publish\n```\n\n## Testing\n\n### Sidekiq\n\nWhen using sidekiq_publisher directly with Sidekiq workers, the testing modes\nprovided by Sidekiq are supported.\n\nRequire the `sidekiq_publisher/testing` file. (This should only be done in test!)\n\n```ruby\nrequire \"sidekiq_publisher/testing\"\n```\n\nThis file requires \"sidekiq/testing\" so there is no need to explictly require both.\nNote that by default, Sidekiq sets the test mode to `fake` and stores jobs in a\n`jobs` array for each worker class.\n\nTo have `SidekiqPublisher` continue to insert jobs into a table within tests\ncall `Sidekiq::Testing.disable!`.\n\n### ActiveJob\n\nWhen using the sidekiq_publisher adapter for `ActiveJob`, use the `ActiveJob`\ntest adapter if you want to run jobs inline during tests.\n\n## Development\n\nAfter checking out the repo, run `bin/setup` to install dependencies. Then,\nrun `rake spec` to run the tests. You can also run `bin/console` for an\ninteractive prompt that will allow you to experiment.\n\nTo install this gem onto your local machine, run `bundle exec rake install`.\n\nTo release a new version, update the version number in `version.rb`, and then\nrun `bundle exec rake release`, which will create a git tag for the version,\npush git commits and tags, and push the `.gem` file to\n[rubygems.org](https://rubygems.org).\n\n## Contributing\n\nBug reports and pull requests are welcome on GitHub at\nhttps://github.com/ezcater/sidekiq_publisher.\n\n## License\n\nThe gem is available as open source under the terms of the\n[MIT License](http://opensource.org/licenses/MIT).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fezcater%2Fsidekiq_publisher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fezcater%2Fsidekiq_publisher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fezcater%2Fsidekiq_publisher/lists"}